date: Wed, 04 Jul 2007 16:21:48 +0100
from: david.parker@metoffice.gov.uk
subject: Re: Fwd: RE: IPCC Table 3.2
to: "Jones, Phil"
Phil
Thanks for both. Steve McIntyre has ashed for the software I used for
calculating the DW statistic so I will send him just the relevant lines.
I use a printout of the Chapter 3 pdf!
Regards
David
On Wed, 2007-07-04 at 14:03 +0100, Phil Jones wrote:
> David,
> Thanks. Tim thinks he could get the procedures to work,
> but won't bother as your explanation is fine. I'll paraphrase it
> and send off. I doubt though that he will be happy. He will
> probably say we didn't give enough explanation. I'll
> be cc'ing to the same list as earlier.
>
> Thanks again.
>
> It would be nice to get the CUP book. I keep opening the pdf
> version of the chapter!
>
> Cheers
> Phil
>
>
> At 12:06 04/07/2007, you wrote:
> >Phil
> >
> >The DW statistic was done on the residuals after removing the AR1
> >persistence as modelled by the restricted maximum likelihood software.
> >That is why the values were close to 2. This procedure is correct
> >because the restricted maximum likelihood software widens the error-bars
> >to take account of AR1; the DW is a test to see whether any further
> >widening is needed, and the results show that it isn't. Steve McIntyre
> >probably used the residuals unadjusted for AR1.
> >
> >I attach our software and 2 sample series. I don't think "R" is used
> >widely here. The DW coding is at lines 78-91 of the plot routine
> >(p_plot_glm_out_annual.pro).
> >
> >I hope this helps
> >
> >Regards
> >
> >David
> >
> >
> >On Wed, 2007-07-04 at 09:08 +0100, Phil Jones wrote:
> > >
> > > David,
> > > Any idea what McIntyre is doing wrong? Maybe check the pv-wave
> > > code and if possible send to me. It is a bit like IDL so I can
> > > probably follow. Maybe there is a routine in PV-Wave for DW.
> > > I've calculated DW on numerous occasions over the past 30 years
> > > for the following:
> > >
> > > 1. In the riverflow reconstruction work. Here I was reconstructing
> > > monthly
> > > flows from rainfall. In the southeastern parts of Britain where there
> > > is
> > > a high groundwater component, rainfall errors propagate for up to a
> > > year.
> > > Here I got some low DWs with values between 0.5 and 1.5.
> > >
> > > 2. In reconstruction from paleo data we calculated DW and
> > > occasionally
> > > got values down to the range 1.2 to 1.5.
> > >
> > > 90% of the time the values were between 1.8 and 2.4 which is where
> > > ours
> > > are in the Tables.
> > >
> > > I've never been able to get values down to the 0.27 and 0.49 that
> > > McIntyre
> > > is talking about.
> > >
> > > Tim Osborn thinks he would be able to run your pv-wave script here,
> > > so
> > > if you can send this with one or two of the series - need the
> > > residuals,
> > > we can check things out.
> > >
> > > Also looking at his R script, I don't think DW can have a p-value as
> > > such,
> > > as it has the two critical levels I was talking about in my response.
> > >
> > > What I think you have calculated DW based upon is the residuals,
> > > so the difference between the original series and the fitted line.
> > >
> > > If anyone at MOHC is adept at R, it would be very nice to point out
> > > what is
> > > wrong with his script !!!
> > >
> > > Cheers
> > > Phil
> > >
> > >
> > >
> > > > X-YMail-OSG:
> > > > Lbmzx_0VM1nx78zXgGstXLvBNj_1bKKWu3u3EfY0YVl7XZyftROTuk6Cnr.ZURgyAw--
> > > > From: "Steve McIntyre"
> > > > To: "'Phil Jones'"
> > > > Subject: RE: IPCC Table 3.2
> > > > Date: Tue, 3 Jul 2007 13:03:18 -0400
> > > > X-Mailer: Microsoft Outlook, Build 10.0.2627
> > > > X-UEA-Spam-Score: 0.1
> > > > X-UEA-Spam-Level: /
> > > > X-UEA-Spam-Flag: NO
> > > >
> > > > Dear Phil, thanks for the prompt reply, but there are a number of
> > > > points that remain very unclear. I am extremely familiar with the
> > > > Durbin-Watson statistic as it is familiar to all econometricians. I
> > > > use the R language which has a convenient Durbin-Watson test in the
> > > > lmtest package (the dwtest function.) When I ran a Durbin-Watson
> > > > test on residuals from fitting a trend, I obtained a Durbin-Watson
> > > > statistic of 0.49 for an OLS-fitted trend to the HadSST2 series
> > > > presently online (over 1850-2005).When I re-fitted a trend line
> > > > using the reported slope of 0.038 deg C/decade, I obtained an even
> > > > lower Durbin-Watson statistic of 0.27. For this same situation,
> > > > you reported a DW statistic of 2.2. I've attached a script in R.
> > > > You say that"We used the lag-1 autocorrelations to calculate the
> > > > reduced number of degrees of freedom of the residuals." In order
> > > > to get a Durbin-Watson statistic of 2.2, you must have done
> > > > something to the data that is not a typical procedure and which is
> > > > not explained in Diggle. The best guess that I could come up with
> > > > as to what you did was that you might have fitted an AR1 arima model
> > > > to the trend residuals and then calculated a DW statistic for the
> > > > residuals for the arima-fit. However, this is just speculation and
> > > > there is no clue in AR4 as to what was done.
> > > >
> > > > BTW the usual interpretation of the DW test in econometrics is as a
> > > > test for first-order autocorrelation, so the exact meaning of using
> > > > a DW test "after allowing for first-order serial correlation" is by
> > > > no means obvious. Again, if you can direct me to an article
> > > > describing the exact procedure that you used together with its
> > > > statistical properties, I'd appreciate it.
> > > >
> > > > Regards, Steve McIntyre
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Phil Jones [mailto:p.jones@uea.ac.uk]
> > > > Sent: Monday, July 02, 2007 9:59 AM
> > > > To: Steve McIntyre
> > > > Cc: Palmer Dave Mr (LIB); david.parker@metoffice.gov.uk;
> > > > Kevin Trenberth; Susan Solomon; Martin Manning
> > > > Subject: Re: IPCC Table 3.2
> > > >
> > > >
> > > > Steve,
> > > > The Durbin-Watson statistics were in an earlier draft of
> > > > the chapter. They were
> > > > removed simply for space reasons, as none were significant.
> > > > As you can see,
> > > > we also removed the lag-1 autcorrelations as well.
> > > >
> > > > REML comes from Diggle et al 1999 (section 4.5 pp
> > > > 64-68). This reference is
> > > > given at the end of the chapter. The page numbers refer to
> > > > the 1999 edition of the book.
> > > > There is a later one available on Amazon, so the page
> > > > numbers may differ in that
> > > > edition. David Parker programmed the calculations of all
> > > > the trends. As far as I
> > > > know he didn't do this with any specific statistical
> > > > packages. He likely used PV-WAVE
> > > > which the Hadley Centre used for almost all their analysis
> > > > work. The use of REML
> > > > is discussed in Appendix 3.A.
> > > >
> > > > DW is very simple to calculate. We used the lag-1
> > > > autocorrelations to calculate the
> > > > reduced number of degrees of freedom of the residuals. This
> > > > number was used with
> > > > the DW statistic to estimate the significance. Basically,
> > > > any DW value above
> > > > about 1.8 is not significant. DW Tables are in some
> > > > statistics books. There should
> > > > be two significance values (for any DW value and N, here
> > > > the effective number
> > > > of degrees of freedom). For the lower of these, values
> > > > below would be significant.
> > > > Values above the upper are not significant. For values in
> > > > between nothing can
> > > > be said. We were always above the upper value. For random
> > > > numbers, the
> > > > DW statistic should return a value about 2. There are
> > > > different sets of Tables for
> > > > different significance levels (1%, 5% etc). We used 5%,
> > > > which is generally
> > > > the one given in text books.
> > > >
> > > > Any statistical package would likely not use the reduced
> > > > number of degrees
> > > > of freedom (reduced based on the lag-1 autocorrelation)
> > > > when giving the
> > > > significance of DW. Using the reduction to the degrees of
> > > > freedom makes the
> > > > test harder to pass.
> > > >
> > > > Phil
> > > >
> > > >
> > > > At 17:36 29/06/2007, Steve McIntyre wrote:
> > > > Dear Phil,
> > > > In Table 3.2 of IPCC AR4, you refer to Durbin-Watson
> > > > statistics for various trend calculations, but do not show
> > > > them. Could you please provide me with these statistics.
> > > >
> > > > I am unfamiliar with any prior use of the Durbin-Watson
> > > > statistic ´after allowing for first-order serial
> > > > correlationĦ. Could you please provide me your statistical
> > > > reference showing how one calculates a Durbin-Watson
> > > > statistic ´after allowing for first-order serial
> > > > correlationĦ and giving significance levels for the
> > > > statistic ´after allowing for first-order serial
> > > > correlationĦ.
> > > >
> > > > Could you please identify the statistical packages used in
> > > > your calculation of REML trends and Durbin-Watson
> > > > statistics?
> > > >
> > > > Would it be correct to say that (1) fitted a trend to the
> > > > various series; (2) fitted an AR1 arima model to the
> > > > residuals from (1)? (3) carried out a Durbin-Watson test on
> > > > the residuals from (2)?
> > > >
> > > > Where applicable, these requests are made under FOI
> > > > provisions.
> > > >
> > > > Thank you for your attention, Steve McIntyre
> > > >
> > > >
> > >
> > >
> > > Prof. Phil Jones
> > > Climatic Research Unit Telephone +44 (0) 1603 592090
> > > School of Environmental Sciences Fax +44 (0) 1603 507784
> > > University of East Anglia
> > > Norwich Email p.jones@uea.ac.uk
> > > NR4 7TJ
> > > UK
> > >
> > ----------------------------------------------------------------------------
> >
> > >
> >
> >Prof. Phil Jones
> >Climatic Research Unit Telephone +44 (0) 1603 592090
> >School of Environmental Sciences Fax +44 (0) 1603 507784
> >University of East Anglia
> >Norwich Email p.jones@uea.ac.uk
> >NR4 7TJ
> >UK
> >----------------------------------------------------------------------------
> >
> >
> >--
> >David Parker Met Office Hadley Centre FitzRoy Road EXETER EX1 3PB UK
> >E-mail: david.parker@metoffice.gov.uk
> >Tel: +44-1392-886649 Fax: +44-1392-885681 http://www.metoffice.gov.uk
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
> Prof. Phil Jones
> Climatic Research Unit Telephone +44 (0) 1603 592090
> School of Environmental Sciences Fax +44 (0) 1603 507784
> University of East Anglia
> Norwich Email p.jones@uea.ac.uk
> NR4 7TJ
> UK
> ----------------------------------------------------------------------------
--
David Parker Met Office Hadley Centre FitzRoy Road EXETER EX1 3PB UK
E-mail: david.parker@metoffice.gov.uk
Tel: +44-1392-886649 Fax: +44-1392-885681 http://www.metoffice.gov.uk