cc: Melissa Free , Peter Thorne , Dian Seidel , Tom Wigley , Thomas R Karl , John Lanzante , Carl Mears , "David C. Bader" , "'Francis W. Zwiers'" , Frank Wentz , Leopold Haimberger , "Michael C. MacCracken" , Phil Jones , Steve Sherwood , Steve Klein , 'Susan Solomon' , Tim Osborn , Gavin Schmidt , "Hack, James J." date: Fri, 11 Jan 2008 18:02:54 -0800 from: Karl Taylor subject: Re: Updated Figures to: santer1@llnl.gov Dear all, The upper panel of figure 2 shows the distribution between differences between simulated and observed trends. The lower panel shows the kind of differences we can expect to get by chance alone (i.e., unforced variability), according to ensembles of simulations by individual models. If we had larger ensembles, we would expect the distribution of these intra-ensemble differences to be more nearly symmetrical about zero. By chance the mean of the results is displaced negatively. As Ben mentioned, if he had included run 1 minus run 2, as well as run 2 minus run 1 (and similarly for other pairs), the expected symmetry would be realized, but he was afraid that this would constitute "double counting". The point of the diagram, however, is to obtain our best estimate of the differences we can expect to get by chance within a model ensemble. I contend that the likelihood of getting a difference of x is equal to the likelihood of getting a difference of -x (within a single model's ensemble), so why not use this information to fill in the pdf in a reasonable way. Thus, I would like to see each difference plotted twice, once with a positive sign and again with a negative sign (and, if you like, we can say we are weighting each point by a half, but of course that doesn't matter here). In this way we will provide a better picture of the true range of differences we would expect to get from each model ensemble. One of the unfortunate problems with the asymmetry of the current figure is that to a casual reader it might suggest a consistency between the intra-ensemble distributions and the model-obs distributions that is not real (and would be unexpected): namely that the differences between trends in runs by individual models also typically are displaced negatively, just like the difference between model and obs. This is, of course, incorrect, and I think we should guard against this misinterpretation. Ben and I have already discussed this point, and I think we're both still a bit unsure on what's the best thing to do here. Perhaps others can provide convincing arguments for keeping the figure as is or making it symmetric as I suggest. There are a few other minor points concerning figure 2 which I'll write down here, so that I don't forget them before I see Ben next on Monday. 1. In panel A, I would plot the histogram for model-obs, not obs-model. I'm used to thinking of errors as being positive when the model value is greater than observed and vice versa. 2. It would appear that if we believe FGOALS or MIROC, then the differences between many of the model runs and obs are not likely to be due to chance alone, but indicate a real discrepancy. If, on the other hand, we believe several of the other models (e.g., MRI or PCM), relatively few of the the model-obs differences are significant. This would seem to indicate that our conclusion depends on which model ensembles we have most confidence in. Am I reasoning this correctly? One complicating factor here is that the normalized differences are ratios, which in the intra-ensemble case roughly measure the amplitude of variability on 20-year time scales (since the true forced trend is the same for both runs) relative to unforced variability on shorter trends (as represented by the standard errors calculated from the de-trended time-series). Thus, a model that has the total variability about right will not yield the correct distribution unless the ratio of the longer-term to shorter-term variability is correct. Similarly, a model that has the incorrect total variance might yield a better normalized trend distribution if the fraction of the total variability exhibited on 20-year time-scales is correct. 3. Instead of '"Between realization" tests', wouldn't it be better to say 'Intra-ensemble tests'? 4. Instead of "Model-vs-model results", wouldn't it be better to say "Realization-vs-realization", not to imply that one model's run is compared to a different model's run. 5. The model labels could be placed as axis labels in place of model number. Best wishes, Karl Ben Santer wrote: > My apologies. I forgot to attach the Figures in my last email. Figures > are appended now. I plead Douglass-induced forgetfulness... > > Best regards, > > Ben > Ben Santer wrote: >> Dear folks, >> >> Here are the revised Figures 1-3 of our contribution to IJoC. >> >> Changes made: >> >> Figure 1: In panel A, I've added some space to separate the UAH and >> RSS trends from the tick marks on the right hand side of the plot, as >> per Leo's request. >> >> Figure 2: As Peter suggested, I've converted the Figure from one to >> two panels. I agree that this is an improvement. The original Figure >> was fairly "busy". Furthermore, the colored symbols (which denote >> results for the "between realization" trend tests) bore no >> relationship to the "Frequency of occurrence" scale on the y-axis. >> This is now clear from panel B. >> >> Figure 3: As Mike suggested, I've removed the legend from the interior >> of the Figure (it's now below the Figure), and have added arrows to >> indicate the theoretically-expected rejection rates for 5%, 10%, and >> 20% tests. As Dian suggested, I've changed the colors and thicknesses >> of the lines indicating results for the "paired trends". Visually, >> attention is now drawn to the results we think are most reasonable - >> the results for the paired trend tests with standard errors adjusted >> for temporal autocorrelation effects. >> >> Please let me know if you would like me to make any other changes. >> >> With best regards, >> >> Ben >> ---------------------------------------------------------------------------- >> >> Benjamin D. Santer >> Program for Climate Model Diagnosis and Intercomparison >> Lawrence Livermore National Laboratory >> P.O. Box 808, Mail Stop L-103 >> Livermore, CA 94550, U.S.A. >> Tel: (925) 422-2486 >> FAX: (925) 422-7675 >> email: santer1@llnl.gov >> ---------------------------------------------------------------------------- >> > >