cc: "Phil Jones" date: Wed, 26 Mar 2008 12:32:16 +0000 from: Ian Harris subject: Re: further CRUTS2.1 vs 3.0 comparisons to: t.osborn@uea.ac.uk Hi Tim, No criticism inferred! Sorry if I was snappy - as you can imagine I'm heartily sick of this dataset! In a professional way of course. My proposal is as follows: • Derive TMN/TMX from gridded DTR and TMP anomalies. Special cases: • Where TMP is missing, TMN/TMX/DTR marked missing. • Where DTR is missing, TMN/TMX = TMP, DTR = 0. Please agree or adjust these (I've probably misunderstood somewhere), then I'll code up a converter later today. Cheers Harry On 25 Mar 2008, at 9:17, Tim Osborn wrote: > Hi Harry, > > sorry for any implied criticism of you, Harry -- I appreciate the > difficulties in working from incomplete documentation (having just > been > modifying ClimGen over the last week!). Just trying to get through > all > this fast, since some partners have collaborators arriving TODAY > and need > the observational data to calibrate their hydrological model with. > > The preference for gridding TMP and DTR and then deriving TMN and TMX > arises in regions where only TMP exists or early on when only TMP > exists. > Suppose one such month has a TMP ANOMALY of -4 C. With no DTR, TMN > or TMX > data, gridding each of these would lead to them being relaxed to their > climatological normals, i.e. anomalies of 0 C. Yet users will ask, > how > can both TMN and TMX have zero anomaly at the same time that TMP > anomaly > is -4 C?!!! The answer might be "well look at the station coverage > data > and you'll see that we don't in fact know what TMN and TMX are, so > just > don't use them at all in this instance". But the alternative would > be to > say that our best guess in the absence of real information is that > TMN and > TMX anomalies are also -4 C, which is what we'd get if we derived them > from TMP and DTR=zero. Not likely to be completely correct, but > the guess > will have some skill over assuming their anomalies are zero. > > I'm not sure what you want to do for CRU TS 3.0, but for ClimGen I > shall > go ahead with TMP, DTR as provided, and then calculate TMN and TMX as > described earlier. > > Cheers > > Tim > > On Sun, March 23, 2008 10:53 pm, Ian Harris wrote: >> Hi, >> >> I did indeed follow a different process from 2.10. This was >> remiss, but >> can be explained. >> >> Firstly, I don't really think the published papers reflect the >> actualité of how the dataset was produced. Neither do the read me >> files. They contain elements of truth but there are glaring >> inaccuracies (including the gridding method, possibly the most >> important single issue). For this reason I've not been as slavish to >> them as I should have been. >> >> Secondly, the tmin and tmax databases are very good. I've removed >> many >> duplicates and 'badly augmented' stations and the two databases I >> used >> are completely aligned. I think that if stations are measuring a >> parameter then that's the truth of the matter (conceding the >> complications arising from measurement times). >> >> There are climatologies for DTR, TMN and TMX. This also pushed me >> towards treating them as primary parameters. >> >> I derived a DTR database from the TMN and TMX databases, and gridded >> all three. That's what you're playing with. >> >> If you want TMN and TMX to be derived from TMP and DTR, so be it >> (though it seems superfluous since it's simply derived). >> >> The problem comes when one considers this process - when we are >> talking >> about TMN and TMX as theoretical constructs that may not have existed >> at all. >> >> The TMN, TMX (and therefore DTR) databases each have 13654 stations. >> Even if they require further cleaning, and I'm sure that they do, >> that's still a lot more than TMP. >> >> Incidentally, sorry but I don't really follow this deduction: >> >>>> Since TMN and TMX are both correlated with TMP, the v2.1 method is >>>> clearly the right way to go. >> >> When either are correlated with TMP, then the strength of that >> correlation is of great interest, isn't it? But the approach of TMN = >> TMP - DTR/2, TMX = TMP + DTR/2 just creates dummies with a >> correlation >> of 1.0. And although the correlation between TMP and DTR is very >> important, DTR is not the only information contained in the TMN >> and TMX >> databases. >> >> Cheers >> >> Harry >> >> On 21 Mar 2008, at 18:17, P.Jones@uea.ac.uk wrote: >> >>> >>> Tim, >>> As far as I'm concerned, the T fields that get gridded >>> are Tmean and DTR. It might be that the climatology >>> is for Tmean, Tx and Tn. If this is the case then we >>> need a DTR climatology. >>> >>> Then what you say below is how I would calculate as you say. >>> >>> Hopefully this is what Harry has done. >>> >>> It might be that Harry has done a Tx and Tn gridding as the >>> two datasets aren't exactly the same - so you lose a bit of >>> data getting DTR, but it is very, very small. >>> >>> Cheers >>> Phil >>> >>> >>> Phil -- if Harry says "yes", then will I get exactly the "correct" >>>> result if I ignore the TMN and TMX files that Harry's made and >>>> instead make my own from TMP and DTR, using: >>>> >>>> TMN=TMP-0.5DTR and TMX=TMP+0.5DTR? >>>> >>>> I tested this and now get much better correlations with v2.1 for >>>> TMN, >>>> and presumably (not checked yet) for TMX. The standard deviations >>>> are now much more similar too. >>>> Here's the comparison when I make my own TMN from TMP and DTR for >>>> v3.0 and compare with TMN from v2.1. >>>> >>>> Tim >>>> >>>> -------------------------- >>>> Phil & Harry, >>>> >>>> Next problem! : >>>> >>>> I've now made comparisons for TMN, TMX and DTR for v3.0 vs. v2.1. >>>> >>>> Attached is the result for TMN. You'll see we now have 4 plots per >>>> month, for Jan-Apr-Jul-Oct. Plot 1 is the temporal >>>> correlation. Plot 2 is the SD of v2.1. Plot 3 is the ratio of >>>> v2.1 >>>> SD to v3.0 SD. Plot 4 is the SD of v3.0. >>>> >>>> Clearly the correlation is rather weak in many areas and the >>>> ratio of >>>> standard deviations exceeds 1.5 across most of S America, Africa >>>> and >>>> India, plus Greenland and N. Russia. >>>> >>>> I think that this may be due to the way you've made the data. I've >>>> checked Mitchell and Jones (2003) and it says very clearly that TMN >>>> (and TMX) are derived variables, taken entirely from the grids >>>> of TMP >>>> and DTR (presumably TMN = TMP - 0.5*DTR and TMX = TMP + >>>> 0.5*DTR???). >>>> >>>> Now if you haven't followed this approach for making v3.0 but have >>>> instead "independently" made gridded fields of TMP, DTR, TMN and >>>> TMP, >>>> relaxing to climatology where there are no nearby observations, so >>>> that they are all primary variables and none are derived, then >>>> if you >>>> have TMP data but not DTR, TMN or TMX values, the latter 3 will be >>>> relaxed to climatology. But the v2.1 methodology would relaxed DTR >>>> to climatology, but TMN would be TMPactual - 0.5*DTRclimatology, >>>> and >>>> hence would still have variations that paralleled the observed >>>> variations in TMPactual. >>>> >>>> Since TMN and TMX are both correlated with TMP, the v2.1 method is >>>> clearly the right way to go. The only time when something >>>> different >>>> to both approaches might be useful is if you have lots of >>>> stations/months with only TMN or TMX but not both. But Harry says >>>> that you generally have both or none. In which case v2.1 will >>>> be the >>>> best we can do. >>>> >>>> Harry -- now that I've confirmed what Mitchell & Jones did to make >>>> v2.1, can you confirmed that you have made v3.0 in the way I >>>> described >>>> above? >>>> >>>> I'll attach the plot with the next >>>> email (too big for this one!). >>>> >>>> Given the time and effort I've put in to CRU TS 3.0, I shall expect >>>> to be a co-author when the paper describing CRU TS 3.0 is written! >>>> >>>> Cheers and happy Easter, >>>> >>>> TimDr Timothy J Osborn, Academic Fellow >>>> Climatic Research Unit >>>> School of Environmental Sciences >>>> University of East Anglia >>>> Norwich NR4 7TJ, UK >>>> >>>> e-mail: t.osborn@uea.ac.uk >>>> phone: +44 1603 592089 >>>> fax: +44 1603 507784 >>>> web: http://www.cru.uea.ac.uk/~timo/ >>>> sunclock: http://www.cru.uea.ac.uk/~timo/sunclock.htm >>>> >>>> >>> >>> >> >> > > > > Ian "Harry" Harris Climatic Research Unit School of Environmental Sciences University of East Anglia Norwich NR4 7TJ United Kingdom