cc: a.watkinson@uea.ac.uk, m.hulme@uea.ac.uk, t.osborn@uea.ac.uk, i.lorenzoni@uea.ac.uk date: Wed, 23 Jan 2008 11:42:59 +0000 from: Saffron O'Neill subject: Re: [Fwd: Journal of Applied Ecology JAPPL-2007-00728] Dear all Thanks for your comments on this. Resubmission to the Journal of Applied Ecology looks likely. It would be good to all meet at the same time to go over the comments, so we can decide which comments need serious adjustments in the text and subsequently also which sections can be removed. I am busy for the next few weeks, but am free 12th Feb am, 13th/14th Feb all day, 15th pm; or 18th/21st all day. Are any of these times convenient? Saffron Saffron O'Neill wrote: > Hi all > > I have just received the review from the polar bear paper. Unfortunately, > it's been rejected. > > It was sent out to 3 reviewers. On the plus side, one is positive and one > is mainly positive with some reservations. The negative review sees merit > in the approach I think, but (as a polar bear expert himself) has a lot of > unanswered questions. > > I'm pretty sure I can address the comments/questions, although it will > take some substantial revision. I'm not really happy about submitting it > as a 'methodological insight' as these papers have to be 5000 words or > under and we struggled to get this version down to the standard length of > 7000 words. > > As I'm about to hand in my thesis and start the MSc job, I wouldn't be > able start doing revisions for a few weeks yet. > > What are your thoughts (resubmission, submission elsewhere?) > > Saffron > > > ---------------------------- Original Message ---------------------------- > Subject: Journal of Applied Ecology JAPPL-2007-00728 > From: japplecol@BritishEcologicalSociety.org > Date: Wed, January 2, 2008 12:43 pm > To: s.o-neill@uea.ac.uk > Cc: japplecol@BritishEcologicalSociety.org > -------------------------------------------------------------------------- > > 02-Jan-2008 > JAPPL-2007-00728 > Expert assessment of the uncertainties of polar bear population dynamics > under climate change > by O'Neill, Saffron; Osborn, Tim; Hulme, Mike; Lorenzoni, Irene; > Watkinson, Andrew > > Dear Miss O'Neill, > > Thank you for submitting the above manuscript to Journal of Applied > Ecology. I have now received referees' reports. > > Two of the reviewers are generally positive, but suggest that you should > look more broadly at the literature on expert opinion in conservation. > Importantly, Reviewer 1 suggests that you should be more clear about the > broader relevance of the technique that you are using, focussing on the > methodological component of the paper - this is necessary in order for the > paper to be eligible for publication in this journal, as we can only > publish papers of broad management relevance, and with some ecological > content. You might consider whether the paper would be better classified > as a Methological Insight rather than a Standard paper? > > Reviewer 2, on the other hand, raises serious issues about the objectives > of the study, which he finds poorly defined. Particularly seriously, he > questions whether this lack of clarity in the objectives feeds through > into the results you have obtained. This issue clearly must be thoroughly > addressed before the paper can be acceptable for publication. > > Since a new manuscript addressing these concerns would be much altered, it > would need to be re-reviewed. In these circumstances, it is Journal policy > to reject the current manuscript but to invite resubmission once the > problems have been resolved. This carries no commitment to eventual > publication but provides an opportunity for re-evaluation by this Journal. > > I am sorry not to be more positive at this stage but please let me know if > you wish us to consider a new version. Any resubmission should be > submitted to the website > (http://mc.manuscriptcentral.com/jappl-besjournals ) as a new manuscript > within four months of this message. Should you decide to take this option, > you should address the referees' comments in full in the new version of > your manuscript. At resubmission, you should upload a file explaining how > you have responded to the various criticisms and including a > point-by-point description of how you have addressed the various comments. > > Once you have revised your manuscript, go to > http://mc.manuscriptcentral.com/jappl-besjournals and login to your > Author Center. Click on "Manuscripts with Decisions," and then click on > "Create a Resubmission" located next to the manuscript number. Then, > follow the steps for resubmitting your manuscript. > > I understand that you may be disappointed with this response, but we wish > you success with this manuscript in whatever course you decide to take. > > Yours sincerely, > > Dr E. J. Milner-Gulland > Editor > Journal of Applied Ecology > > *************************************************** > COMMENTS FROM ASSOCIATE EDITOR: Dr E. J. Milner-Gulland > > Associate Editor: 1 > Comments to the Author: > (There are no comments) > > > > *************************************************** > Referees' comments: > > Attached files - please note that where a referee has uploaded comments as > an attachment they will not be copied below. The attachment can be found > in your Author Centre on Manuscript Central by clicking on 'Manuscripts > with decisions' in the 'My manuscripts' list. Locate the correct file in > the table and click on 'view decision letter' in the Status column. A copy > of this letter will display. Scroll to the end of the letter to see a list > of 'Files Attached'. Click on the filename to view the attachment. > > Reviewer: 1 > REPORT FOR THE AUTHORS FROM REFEREE > This paper is an interesting and well-thought out study addressing a > topical issue, determining the potential impacts of climate change on > species. It demonstrates clearly the application of a relatively novel > approach to surveying expert opinion within ecology and that considerably > improves upon the existing methodologies being used. > > Results provide important information for the ecological and wider > community about future polar bear dynamics, and about the importance of > not just relying on statements from single experts or studies. > > My main concern was that the paper does not properly address how such an > approach can or should be used for other applications in ecology. The > authors state ‘It is imperative that expert opinion is elicited through a > systematic and thorough methodology’ but then ‘as portrayal of polar bear > risk in the media is informed by expert opinion’. > > Should such an approach be applied when there is not media interest? For > other species? How might use of this methodology help managers make better > decisions then the current informal methods being used? These questions > should be addressed. > > The manuscript would, I think, be considerably strengthened by inclusion > of a paragraph explicitly stating some guidelines and recommendations for > adapting this style of survey for use with other species or issues. I > would also like to see some more general references to the literature on > expert judgement included (eg. Morgan, 1990, O’Hagan et al 2006, Kaplan > 1999, Walker 2001) and to other studies that have previously used or > addressed the use of expert opinion in ecology (eg. Akcakaya 2000, Martin > 2005, White 2005, …..). The references to expert surveys listed currently > are almost all taken solely from the climate change literature. > > The current terminology used to describe the different types of > uncertainty (eg. between and within expert) is confusing and should be > revised – perhaps more in line with the delineations of uncertainty like > ‘variability’ and ‘incertitude’ used in other studies? (also see comment > in minor points below) > > My general feeling was also that the sections on ‘sea ice’, ‘population > modelling’ and ‘expert surveys’ in the introduction could all be > tightened. Possible suggestions are shortening the 2nd and 3rd paragraphs > of the section on ‘Sea ice and relationship with polar bear ecology’, as > they do not add substantially more to the paper, the mention of contingent > and non-contingent phenomena seems unnecessary, and as most readers will > be familiar with PVA and its limitations, the more relevant parts of the > paragraph on PVA could just be incorporated into the following paragraph. > > > Minor points: > > Page 6, line 22. The authors state’ there are dangers in confusing > knowledge with opinion’. This is true, but not in itself a justification > for using a ‘systematic methodology’ for surveying opinion. The reasons > for using an approach that is (i) systematic, (ii) quantitative and (iii) > involves multiple experts should be explicitly stated. > > Page 6: line 25 append sentence to: ‘…expert opinion about future polar > bear population dynamics. ‘ > > Page 7, para 1: ’ I think this paragraph would read better if the > sentence beginning ‘Expert opinion is of value…’ was placed at the > beginning of the paragraph. For the first sentence, append to ‘…define the > current knowledge and range of uncertainty…The meaning of the second > sentence is also not clear. > > Page 7, para 2: Why are we distinguishing between contingent and > non-contingent phenomena? Explain. Reword the final sentence to > ‘….provide a formal assessment of the current expert opinion on the likely > future…’. > > Page 12, line 13: Use of the term ‘controversial’ here is a bit melodramatic. > > Page 13, line 5: para 2. Include an additional sentence stating what the > Delphi method is, and a reference. > > Page 13, line 24: Replace term ‘controversial’ with ‘not straightforward’ > or a similar phrase. > > Page 19, para 3: The descriptions given between expert and within expert > uncertainty are unclear. Remove the word ‘expression’. Replace with > something more like ‘result of the different scientific understandings of > the individual experts.’ > > In fact, simply referring to the ‘variability between experts’ rather then > ‘between expert uncertainty’ may actually be easier for readers to follow. > The description of within expert uncertainty is likewise very unclear. > > > Page 20, line 1: Include an example in brackets to clarify the type of > ‘simpler’ aggregation methods you are referring to. Eg. Mean, median, etc. > Replace ‘perform better’ with ‘as least as well’. > > Page 20, line 17: ‘The large uncertainties illuminate the difficulties > associated with using expert’. > > No they don’t. The variability or ‘between expert’ uncertainty does > partly, but models and other approaches produce similar ranges of > uncertainty when there is minimal data available. > > The phrase ‘derives from the expression of the scientific understanding of > the individual experts’ > > To ease in comparisons in Figures 4-6, it would be useful to superimpose a > line representing the median average judgment [as in the last box plot] > onto the box plots of the other experts. > > It would also be good to comment on the expert’s perceptions of their own > expertise. Were there any trends between perceived expertise and the type > of responses given? > > The results are particularly compelling when restated as 90% confidence > intervals – ie. the experts were all apparently at least 90% confident > that polar bear populations would not be extinct by 2050, and in most > cases 90% confidence they would not increase etc. > > > References > > Akcakaya, H. R., S. Ferson, et al. (2000). "Making consistent IUCN > classifications under uncertainty." Conservation Biology 14(4): 1001-1013. > > Kaplan, S. and D. Burmaster (1999). "C. How, When, Why to Use All of the > Evidence > doi:10.1111/j.1539-6924.1999.tb00388.x." Risk Analysis 19(1): 55-62. > > Martin, T. G., P. M. Kuhnert, et al. (2005). "The power of expert opinion > in ecological models using Bayesian methods: Impact of grazing on birds." > Ecological Applications 15(1): 266-280. > > > Morgan, M. G. and M. Henrion (1990). Uncertainty: a guide to dealing with > uncertainty in quantitative risk and policy analysis. New York, Cambridge > University Press. > > O'Hagan, A., C. E. Buck, et al. (2006). Uncertain judgments : eliciting > expert probabilities. West Sussex, John Wiley and Sons Ltd. > > Walker, K. D., J. S. Evans, et al. (2001). "Use of expert judgment in > exposure assessment - Part I. Characterization of personal exposure to > benzene." Journal of Exposure Analysis and Environmental Epidemiology > 11(4): 308-322. > > White, P. C. L., N. V. Jennings, et al. (2005). "Questionnaires in > ecology: a review of past use and recommendations for best practice." > Journal of Applied Ecology 42(3): 421-430. > > Reviewer: 2 > REPORT FOR THE AUTHORS FROM REFEREE > Review of O’Neill et al. “Expert assessment of the uncertainties of polar > bear population dynamics under climate change” > > The Paper: > The aim of this paper (p7) is to provide an expert assessment of the > future of polar bears under climate warming. > > The assessment is derived from a survey of 10 “experts” in polar bear > biology. > > Participants in the survey were asked to project the future for polar > bears in 5 different geographic regions. Future projections were supposed > to be based upon maps depicting the future sea ice extent and temporal > availability which were provided. > > Results were expressed as the range of values projected by the array of 10 > “experts”. > > General Comments: > > The uneven quality of data available worldwide and the absence of > statistical analyses which could be applied over broad ranges of polar > bear habitat makes the use of expert judgment a valuable tool in > understand the probable futures of polar bears across their range. > Hence, a survey of the judgments of a variety of polar bear experts has > merit as a way to understand where polar bears may be headed as > availability of their sea ice habitat declines. Whereas this approach has > some merit, many factors serve to diminish the value of the outcomes > reported here. Here, I list a couple of general comments. Then, I > provide a few comments specific to elements in the text of this MS. > Hereafter, I refer to this MS as “O’Neill et al”. > > 1. My biggest general concern about O’Neill et al is that there seems to > be an unclear objective which confuses the ability to interpret the > results. It is not clear whether the goal of O’Neill et al. is to > establish how much variation there is in the range of judgments that > different experts might make given a certain set of conditions-the > “specified climate future (line 8 page 8)”, or whether it is to project, > to the extent practicable with expert judgment, the future of polar bears > (line 1 page 8). These are quite different things. There will be > variation among the judgments of experts in both, but the goals are quite > different, and the information provided to the experts as well as the > questions asked would be very different. The pretext of this project > seems to be that O’Neill et al are interested in resolving the apparent > discrepancy between what may be viewed as an alarmist general media > impression of impending disaster for polar bears, a current (as opposed to > a projected) worldwide population which may be in pretty good shape, and > views of at least one global warming naysayer. To address these seemingly > different perspectives, it would be good to know how the judgments of > experts fall out regarding the future. O’Neill et al state on line 23, > page 3, that how various narratives represent the range of views of the > expert community (on the present and likely future status of polar bears) > is their goal. Conversely, they express concern that one participant > (line 22, page 19) may have used other information. If the goal of > O’Neill et al is indeed to project the future for polar bears, the > participants need different instructions. They cannot do this given the > limited information which the O’Neill et al elicitation provided. From > that information, they can only meet the goal of assessing the degree to > which the participants may differ given the same information. The range > of expert views on the future of polar bears is an interesting topic, and > appears to be the goal. The range of views on how a specified future may > affect polar bears is much less interesting and although it seems that is > what O’Neill et al have set their participants up for, it does not address > the perceived discrepancy described in the introduction (page 3:17-21) > 2. Related to #2, this MS fails to take into account the observational > record of sea ice. The participants were asked to project based upon a > specific scenario of temporal/spatial sea ice change, but they were not > explicitly provided any of the data regarding how the ice has changed in > the observational period. Those changes and the observed effects of those > changes are an important part of “judging” what future changes are likely > to be given the range of values projected by climate models. Again, if > the future given a specified future is the goal that is one thing, but if > O’Neill et al are after something like what experts think may be the real > future; that is very different and requires some relevant information from > the past and present. For example, the O’Neill et al defined > “Archipelago” includes Baffin Bay and much of Davis Strait. The > projections in the O’Neill et al scenario provided show little ice change > in these regions despite very great changes there in recent years. In > fact most climate models have a problem projecting the sea ice in Davis > strait, and most show more ice remaining in that region in 100 years than > we have there now. Most of the experts do not know that, but if they were > told that, it might alter how they responded to the sea ice scenarios > presented. If the “experts” are to “judge” what the future may bring to > that region, they need some context, not just one view of modeled future > sea ice. In general, the fact that observed changes in the sea ice have > been faster than forecasted must be a part of any effort to project the > future of polar bears. But, it was not included here. > 3. In the discussion, O’Neill et al appear to view their results as if > they have addressed the real future for polar bears rather than some > specified future. I don’t think they have. Hence, my biggest concern is > that I don’t think the approach outlined here has gotten O’Neill et al to > where they think they are. As it is, this MS describes a methodology > which could be useful if conducted a bit differently. It does not, > however, provide the best judgments of the experts regarding the likely > future for polar bears. > > Specific Comments: Here, I make a limited number of specific comments > regarding the text of O’Neill et al. > 1. Given the nature of the survey and the expectation that the experts > comment on a “specified” sea ice future, the title of O’Neill et al should > be something like “An approach to using expert knowledge to assess > uncertainties in future polar bear populations”. This MS expresses the > gist of how such a survey might be conducted, but it doesn’t really > provide a sense of the range of expert judgments of what the real future > might be. > 2. Due to the rapidly changing information base (including updated climate > model outputs) any MS like this one needs to specify clearly the date when > it was compiled. Else, it will just contribute to confusion such as the > confusion raised by Lomborg and others when they claim polar bear > populations have grown and are not threatened by declining sea ice. > Lomborg confuses the fact that protections from over hunting which went > into effect beginning in the late 60’s and early 70’s, allowed bear > numbers to grow into the 90’s; with the fact that declining spatiotemporal > availability of sea ice, polar bear security now is, in at least some > regions, threatened. Without clearly specifying time frames of reference, > similar confusion could result form a result such as this. Most polar > bear experts, for example, would have a more dire assessment of the future > for polar bears now than they did only a year ago. > 3. On page 2:10-12 of the MS, O’Neill et al state that “Negative impacts > of climatic warming on polar bears have been suggested, but cannot be > quantified as no models yet exist to analyze the relationship between > polar bear population dynamics and climate change”. It is not clear to me > what constitutes “models” in this context, but detailed data and analyses > relating specific changes in life history traits of polar bears to > changing sea ice conditions, were available for the Hudson Bay region when > O’Neill et al was in preparation. > 4. Page 3:8-9, where in the paper is the foundation of this statement. > Whereas I and most polar bear experts would agree, this seems to be a > stand-a-lone conclusion of O’Neill et al rather than a conclusion which > can be derived from the survey results. Clarify where this came from. > 5. Page 4:4-7, it is not clear that this paper really addresses either of > these issues. This is a nice statement about critical issues, but they > are not addressed here. > 6. Page 4:15, the perennial sea ice has been declining at around 8.6% per > decade, not sure where this 4.7 came from. > 7. Pages 4 and 5: there is a smattering of biological material here, but > it is limited, incomplete, and misleading. And, it is not used in the > remainder of the document, so not clear why it is here. > 8. Page 5:13, what does the first clause mean? > 9. Page 6-7, I find the description of opinion and judgment in O’Neill et > al confusing at best. Please define more explicitly. Are you after > opinions or expert judgments. I have many opinions, but I try as hard as > I can to keep them out of my judgments. > 10. Page 7 what is this “contingent versus non-contingent phenomena? You > make a big point of what they are here but never refer to them again. > 11. Page 8:1, why are you interested in the future of polar bear > populations in this context. Why not just assess how expert judgments may > differ regarding the future. The “controversy” in the press is a > manufactured controversy created by people like Lomborg, who has been > totally discredited as making statements simply to create doubt > (http://www.lomborg-errors.dk/coolitAchap1.htm). Scientific papers should > be above delving into such controversies. This question is important, and > expert judgments may be a way to address it; regardless of what the > general media or global warming naysayers may say about it. > 12. Page 8:21, no March in Figure 1. > 13. Page 9:23, which GCMs had ensembles of projections? IN general, do > not use terms like most, or several or---, just state the number. In this > paper, there are only 10 participants and there are only a few models, say > how many when you are referring to a limited set of them. > 14. Page 9:25, why did you use these models of the 21 which were available > in the AR-4? What was your selection criterion. > 15. Page10:10, why pick one model. Why not use the mean of your ensemble > to build your projection of sea ice? > 16. page 12: 7-10, what does this mean? Need to restate. > 17. Page 15:1, in view of the extreme range of estimates provided by the > experts, it may have been useful to censor the upper and lower extremes in > each category of answer. Given that polar bear biology is pretty well > known and that their dependence on the sea ice for foraging is absolute, > this sort of range suggests that something is going on other than a > difference in judgment of how much a given decline is likely to affect > bears in a particular area. > 18. Page 17:24-24--page 18:1-3, whereas I agree with this statement, there > is no real support for this statement in the MS. This appears to be a > conclusion of O’Neill et al. It should be stated as such, or cited or > otherwise substantiated. > 19. Page 18:23-25. This harkens back to the difference between polar > bears in a future you specified as opposed to projections based on what we > think the real future may be. See general comments. > 20. Page 19:16-25, this is a problem for this method. I am pretty sure > that each participant imbued his/her responses with other information and > knowledge about sea ice, sea ice projections, and observations to a > greater or lesser degree. In a survey like this, where you tried to get > the participants to judge based only on your inputs, some uncontrollable > ancillary information is going to creep in. This probably contributed > quite a bit to the range of answers you received. > 21. Page 20: 1-7, what does this mean? > 22. Page 20:17, I think the large uncertainties are at least partly due to > the design wherein some participants tried less successfully than others > to limit their judgments to your prescribed future. Surely, there will > always be a range of answers to any question wherein there is uncertainty, > but in this case, I think your range of answers was partly dependent on > different degrees of adhering to your future. Also, you had a huge range > in expertise in your “expert” pool. Finally, you included one expert who > has very publicly taken a political approach to the effects of sea ice > loss on polar bears. > 23. Page 22: 14-18, again, this seems to be a conclusion of O’Neill et al. > However, as written, it is not clear that it is not a conclusion of the > experts polled. Reword this to put it in proper context. > > In conclusion, I find that O’Neill et al describes a potentially useful > methodology. With clarification of the goals, better descriptions of the > methods, and more careful formation of conclusions, this could be a useful > paper. > I hope you find these comments helpful. > > Steven C. Amstrup > > > Reviewer: 3 > REPORT FOR THE AUTHORS FROM REFEREE > Manuscript reference number: JAPPL-2007-00728 > Authors: O'Neill, Saffron; Osborn, Tim; Hulme, Mike; Lorenzoni, Irene; > Watkinson, Andrew > Title: Expert assessment of the uncertainties of polar bear population > dynamics under climate change > > > Expert elicitation is gaining prominence in ecology and natural resource > management as a way of capturing information and the uncertainty around > this information. Using a remote survey Delphi approach, O’Neill et al > elicit information on polar bear dynamics with respect to climate change – > a flagship species in the climate change debate. Of ten experts, most > predicted a significant decline in polar bear population size and range as > a result of climate change, although there was considerable variability > between experts. > > This paper is an excellent demonstration of the power of expert opinion to > inform us on complex ecological interactions, where limited resources and > the need to make urgent conservation management decisions, precludes the > sole reliance on long-term empirical studies. Indeed, with finite > research resources, the skilful elicitation and use of expert knowledge > will play a major component in endangered species conservation and > management in the future. This is a well executed and competent use of > expert knowledge on a topic of considerable public and scientific > interest. > > My only criticism of this paper is that it does not take advantage of > Bayesian methods and make full use of the expert information, informing > empirical data, to produce posterior estimates which capture the > uncertainty in expert knowledge (prior information) and the uncertainty in > the empirical data (likelihood). Perhaps this could form another study. > Recent examples of this approach include Crome et al 1996 and Martin et al > 2005. > > Minor comments > Page 2, line 2. Delete “(the first ever….)”. This is not the first survey > of expert opinion. Perhaps you mean, the first ever taken for polar bears > based on sea-ice data? > > Page 2. line 21, 22. What does decline in population mean? Decline in > population size, number of populations, sub-populations? > > References > Crome, F. H. J., M. R. Thomas, and L. A. Moore. 1996. A novel Bayesian > approach to assessing impacts of rain forest logging. Ecological > Applications 6:1104-1123. > > Martin, T. G., P. M. Kuhnert, K. Mengersen, and H. P. Possingham. 2005. > The power of expert opinion in ecological models using Bayesian methods: > Impact of grazing on birds. Ecological Applications 15:266-280. > > *************************************************** > > Reply to: > > Gillian Kerby > Managing Editor, Journal of Applied Ecology > e-mail:japplecol@BritishEcologicalSociety.org > phone: +44 (0)1780 752833 fax: +44 (0)1780 752833 > > British Ecological Society > 26, Blades Court, Deodar Road > Putney, London SW15 2NU, UK. > phone: +44 (0) 208 8770740 > > *************************************************** > > The Journal of Applied Ecology is published by the British Ecological > Society, a limited company, registered in England No. 1522897, and a > registered charity No. 281213. Information or advice given to members or > others by or on behalf of the Society is given on the basis that no > liability attaches to the Society, its Council members, Officers, or > representatives in respect thereof. > > Journal information and instructions for contributors: > http://www.blackwellpublishing.com/journals/jpe > and www.BritishEcologicalSociety.org > Submissions to: > http://mc.manuscriptcentral.com/jappl-besjournals > > *************************************************** > The Journal of Applied Ecology is published by the British Ecological > Society, a limited company, registered in England No. 1522897, and a > registered charity No. 281213. Information or advice given to members or > others by or on behalf of the Society is given on the basis that no > liability attaches to the Society, its Council members, Officers, or > representatives in respect thereof > >