date: Thu, 10 Jul 2008 10:47:24 -0700 from: Ben Santer subject: [Fwd: JOC-08-0098.R1 - Decision on Manuscript] to: "Thorne, Peter" , Leopold Haimberger , Karl Taylor , Tom Wigley , John Lanzante , ssolomon@frii.com, Melissa Free , peter gleckler , "'Philip D. Jones'" , Thomas R Karl , Steve Klein , carl mears , Doug Nychka , Gavin Schmidt , Steven Sherwood , Frank Wentz Dear folks, I just returned from my trip to Australia - I had a great time there. Now (sadly) it's back to the reality of Douglass et al. I'm forwarding the second set of comments from the two Reviewers. As you'll see, Reviewer 1 was very happy with the revisions we've made to the paper. Reviewer 2 was somewhat crankier. The good news is that the editor (Glenn McGregor) will not send the paper back to Reviewer 2, and is requesting only minor changes in response to the Reviewer's comments. Once again, Reviewer 2 gets hung up on the issue of fitting higher-order autoregressive models to the temperature time series used in our paper. As noted in our response to the Reviewer, this is a relatively minor technical point. The main point is that we include an estimate of the standard error of the observed trend. DCPS07 do not, which is the main error in their analysis. In calculating modeled and observed standard errors, we assume an AR-1 model of the regression residuals. This assumption is not unreasonable for many meteorological time series. We and others have made it in a number of previous studies. Reviewer 2 would have liked us to fit higher-order autoregressive models to the T2, T2LT, and TS-T2LT time series. This is a difficult business, particularly given the relatively short length of the time series available here. There is no easy way to reliably estimate the parameters of higher-order AR models from 20 to 30 years of data. The same applies to reliable estimation of the spectral density at frequency zero (since we have only 2-3 independent samples for estimating the spectral density at frequency zero). Reviewer 2's comments are not particularly relevant to the specific problem we are dealing with here. It's also worth mentioning that use of higher-order AR models for estimating trend standard errors would likely lead to SMALLER effective sample sizes and LARGER standard errors, thus making it even more difficult to find significant differences between modelled and observed trends! Our use of an AR-1 model makes it easier for us to obtain "DCPS07-like" results, and to find significant differences between modelled and observed trends. DCPS cannot claim, therefore, that our test somehow stacks the deck in favor of obtaining a non-significance trend difference - which they might claim if we used a (poorly-constrained) higher-order AR model for estimating standard errors. The Reviewer does not want to "see the method proposed in this paper become established as the default method of estimating standard errors in climatological time series". We do not claim universal applicability of our approach. There may well be circumstances in which it is more appropriate to use higher-order AR models in estimating standard errors. I'd be happy to make a statement to this effect in the revised paper. I have to confess that I was a little ticked off by Reviewer 2's comments. The bit about "wilfully ignoring" time series literature was uncalled for. Together with my former MPI colleague Wolfgang Brueggemann, I've fooled around with a lot of different methods of estimating standard errors, in both the time domain and frequency domain. One could write a whole paper on this subject alone. Such a paper would not help us to expose the statistical deficiencies in DCPS07. Nor would in-depth exploration of this issue lead to the shorter paper requested by the Reviewer. It should take me a few days to revise the paper and draft a response to Reviewer 2's comments. I'll send you the revised paper and draft response early next week. Slowly but surely, we are getting there! With best regards, Ben ---------------------------------------------------------------------------- Benjamin D. Santer Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory P.O. Box 808, Mail Stop L-103 Livermore, CA 94550, U.S.A. Tel: (925) 422-3840 FAX: (925) 422-7675 email: santer1@llnl.gov ---------------------------------------------------------------------------- X-Account-Key: account1 Return-Path: Received: from mail-1.llnl.gov ([unix socket]) by mail-1.llnl.gov (Cyrus v2.2.12) with LMTPA; Tue, 08 Jul 2008 23:07:04 -0700 Received: from mail-1.llnl.gov (localhost.localdomain [127.0.0.1]) by mail-1.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id m69674LM010414 for <[vacation]santer1@mail.llnl.gov>; Tue, 8 Jul 2008 23:07:04 -0700 Received: (from vacmgr@localhost) by mail-1.llnl.gov (8.13.1/8.13.1/Submit) id m69674am010413 for [vacation]santer1@mail.llnl.gov; Tue, 8 Jul 2008 23:07:04 -0700 X-Authentication-Warning: mail-1.llnl.gov: vacmgr set sender to g.mcgregor@auckland.ac.nz using -f Received: from nspiron-2.llnl.gov (nspiron-2.llnl.gov [128.115.41.82]) by mail-1.llnl.gov (8.13.1/8.12.3/LLNL evision: 1.7 $) with ESMTP id m6966u3i010371 for ; Tue, 8 Jul 2008 23:07:03 -0700 X-Attachments: * JOC-production-requ.doc X-IronPort-AV: E=McAfee;i="5200,2160,5334"; a="21123949" X-IronPort-AV: E=Sophos;i="4.30,329,1212390000"; d="doc'32?scan'32,208,32";a="21123949" Received: from nsziron-3.llnl.gov ([128.115.249.83]) by nspiron-2.llnl.gov with ESMTP; 08 Jul 2008 23:07:01 -0700 X-Attachments: * JOC-production-requ.doc X-IronPort-AV: E=McAfee;i="5200,2160,5334"; a="47882084" X-IronPort-AV: E=Sophos;i="4.30,329,1212390000"; d="doc'32?scan'32,208,32";a="47882084" Received: from uranus.scholarone.com ([170.107.181.135]) by nsziron-3.llnl.gov with ESMTP; 08 Jul 2008 23:07:00 -0700 Received: from tss1be0007 (tss1be0007 [10.237.148.33]) by uranus.scholarone.com (Postfix) with ESMTP id B3F62B2883B for ; Wed, 9 Jul 2008 02:06:59 -0400 (EDT) Message-ID: <363254525.1215583619734.JavaMail.wladmin@tss1be0007> Date: Wed, 9 Jul 2008 02:06:59 -0400 (EDT) From: g.mcgregor@auckland.ac.nz To: santer1@llnl.gov Subject: JOC-08-0098.R1 - Decision on Manuscript Errors-To: g.mcgregor@auckland.ac.nz Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1328_363182756.1215583619708" X-Errors-To: g.mcgregor@auckland.ac.nz Sender: onbehalfof@scholarone.com Ben A couple of comments before you read the standard letter below and associated comments. You have seen those of Referee 1. Now the long awaited comments from reviewer 2 are back. The reviewer raises three issues. Of these it would seem to me that length is one issue not to concern yourself with. If you could include a couple of lines clarifying the nature of s(bo) as suggested by the reviewer that would be helpful. Lastly could you address the comment about the time series autocorrelation issue - I think no further anaysis is required here but what is needed is perhaps a short defence of the strategy adopted. I will not send the revised version back to the reviewer but will proceed based on the alterations made to the paper and your associated response summarising how you have addressed the issues raised. Needless to say I look forward to publishing this paper once you have got back to me with the suggested minor alterations completed. ============================ Dear Dr Santer Manuscript # JOC-08-0098.R1 entitled "Consistency of Modelled and Observed Temperature Trends in the Tropical Troposphere" which you submitted to the International Journal of Climatology, has been reviewed. The comments of the referee(s), all of whom are leading international experts in this field, are included at the bottom of this letter. If the reviewer submitted comments as an attachment this will only be visible via your Author Centre. It will not be attached to this email. Log in to Manuscript Central, go to your Author Centre, find your manuscript in the "Manuscripts with Decisions" queue. Click on the Decision Letter link. Within the Decision letter is a further link to the reviewer attachment. I am very happy to grant conditional acceptance of the paper, subject to you making satisfactory revisions as clarified below. These revisions are minor, in the sense that there are no major recalculations or major analyses required, but there may be less-major analyses needed and there are a number of important revisions to the text that are necessary. Please review the attached document listing the file requirements for your revision. In your revisions, please address all of the points made by each reviewer, make suitable edits to the manuscript, and include a point-by-point response to each reviewer comment with your response. Use the space provided or upload a file documenting your responses. I will study your responses and determine if the edits were handled satisfactorily. I would like to receive the revised manuscript as soon as is convenient, but with a deadline about 2 months from now. This should be sufficient time given the minor revisions, but please communicate with me (email is best) now or closer to the time if you need an extension of the deadline. I feel confident that, pending careful revisions as outlined above, I will be able to publish your paper in the International Journal of Climatology. You can upload your revised manuscript and submit it through your Author Centre. Log into http://mc.manuscriptcentral.com/joc and enter your Author Centre, where you will find your manuscript title listed under "Manuscripts with Decisions." IMPORTANT: Please make sure you closely follow the instructions for acceptable files. When submitting (uploading) your revised manuscript, please delete the file(s) that you wish to replace and then upload the revised file(s). Please remember that the publishers will not accept a manuscript unless accompanied by the Copyright Transfer Agreement. Please go to: http://www3.interscience.wiley.com/homepages/4735/nscta.pdf The Copyright Transfer Form and the Permissions Form should be scanned and uploaded with your submission to Manuscript Central, designated as "Supplemental Material not for review". If you do not have access to a scanner, further instructions will be provided upon the acceptance of your paper. Forms should not be sent to the editorial office. Once again, thank you for submitting your manuscript to the International Journal of Climatology. I look forward to receiving your revision. Sincerely, Prof. Glenn McGregor Editor, International Journal of Climatology Referee(s)' Comments to Author: Referee: 1 Comments to the Author I think the authors have responded more than satisfactorily to my comments. I have only a few minor additional comments 1. p19, lines 37-39 (short sentence beginning with "This is essentially ..."). I would leave this out I think. DCPS07 do not clearly articulate the hypothesis that they wished to test, so it is slightly misleading to say that they considered a specific hypothesis. 2. The author revised some uses of the word "yields", but seems to have introduced other instances in the conclusions. This is just a matter of style and personal preference, but for me, saying a calculation yields something makes it sound a bit like the author is pleased with the calculation because it produces the anticipated or desired result. 3. In the acknowledgments, please list my affiliation as "Environment Canada". Francis Zwiers 21 June 2008 ========================== Referee 2 >From my point of view, there were three major issues about the original paper - 1. Its length, 2. The failure to distinguish between the two types of test, 3. The treatment of autocorrelation. On point 1, I still feel the paper is longer than it needs to be, though as far as I can see, I was the only one of the 2 referees + editor to raise that as a serious concern. I appreciate the efforts they have made to remove some tangential material, and it has required some additional writing to incorporate point 2. So, if the editor is happy with that, I'm willing to say OK on that point. On point 2, yes, I'm very satisfied with the work they have done on this. The two types of hypothesis are now clearly distinguished, and the second one is laid out in a way that makes clear both the similarities and differences with the DCPS approach. It's a little odd that the revised test doesn't seem to make any use of the individual-model standard errors, equation (4). Instead, what they do is to compute the sample mean and standard deviation of the estimated model trends, equations (7) and (9) respectively, and then a separate estimate of s{bo}, the standard error of the observational trend. I wondered about this, whether the proposed test would even be correct if the individual-model standard errors were unequal. In fact, if you square both sides of equation (9), the sample variance is an unbiased estimator of the variance of <> even if the individual variances of the model estimates are unequal. So the caveat on page 24, lines 6-9, may not be strictly necessary. I think the test does work (in the form proposed by the authors, not DCPS), but its potential disadvantage is that maybe equation (9) is not actually the most efficient way to estimate the variance of the estimated ensemble mean trend. This might be more of an issue if the authors were trying to do it with a small number of models (3, say), rather than the relatively large 19. The test as proposed is still an order of magnitude improvement on ignoring the observational variance altogether (the DCPS test), but I don't think it's the last word on how to do this kind of inter-model comparison, especially when there are an unequal number of realizations for each model. By the way, this wasn't what my simulation was about. The only point I was trying to make there was to show the potential advantage of taking an ensemble mean viewpoint. That is now clear from the revised paper. There is one point that I only spotted while reading the revision (though it was present in the original version) and I think some clarification from the authors may help. If you read section 4.1, it talks about s{bm} and s{bo} and then it tells you the formula for s{bm}. But there isn't a formula for s{bo}. Of course, s{bo} is calculated exactly the same way as s{bm}, but using the observational rather than the model data. That's obvious to anyone who understands the basic statistics being used here, but since the authors' main point is that DCPS didn't use s{bo} at all, maybe they should take a couple of extra lines to say that explicitly. Regarding the time series autocorrelation issue, I appreciate the authors' putting more work into conducting a sensitivity study where they varied the length of the averaging interval. I have to say, I would have been more impressed if they had conducted a sensitivity study to different formulas for calculating the variance of the trend (such as AR(p) for p>1, or various formulas for approximating the spectral density at frequency zero, such as the Heidelberger-Welsh procedure which is popular in operations research). I think what concerns me about the revision is that there is no mention whatsoever there is a much bigger theory of time series analysis of which the present treatment is very much a special case. For example, what if there is long-range dependence? I'm willing to let go the fact that the proposed method is probably adequate for the particular time series studied in the present paper, but my concern is that I wouldn't want to see the method proposed in this paper become established as the default method of estimating standard errors in climatological time series. The paper as currently written creates the impression that the authors are either unaware of the time series literature or are wilfully ignoring it. As far as a recommendation on the paper is concerned, I'm willing to recommend a provisional acceptance. I would urge the authors to give some further consideration to these points in preparing the final version. Attachment Converted: "c:\eudora\attach\JOC-production-requ4.doc"