A Guest post by Dennis Bray and Hans von Storch

The Survey of Climate Scientists (Bray and von Storch CliSci2008) closed on October 1. On September 29 preemptive negative comments were posted on the popular blog Real Climate by Gavin Schmidt, a NASA scientist. Here we would like to respond to some of the issues that he raised.

First, for the record, as there seems to be some confusion as to who we are, I am a sociologist and Hans von Storch is a climate scientist.

First a thanks to those people who took the time to respond to the survey and those who contributed favorable comments on the RealClimate blog.

I attend first to the blog posting and then some of the responses.

Gavin begins by noting weaknesses in previous surveys.

Problems have ranged from deciding who is qualified to respond; questions that were not specific enough and losing control of who answered. These problems have meant that the results were less than useful.

The Samples

The 1996 Sample

(Note: for a full description of sampling and response rates and results see this PDF)

In 1996 the survey was distributed somewhat differently than the approach that we are now using. Due to privacy considerations the American Meteorological Society, and German and Danish equivalents were sent hard copies of surveys to distribute to a random sample of their members. At a request from the Italian science community, arrangements were made to include Italian climate scientists, the mail out was handled in Italy. Finally, known and respected climate science institutes were employed to increase the list of potential respondents. In all, the survey covered 5 countries, Germany, USA, Canada, Denmark and Italy. Response rate was approximately 30% and if publication in refereed climate science journals and bulletins is an indication, the results were readily accepted by the climate science community. So no apparent problem there. There were however some comments concerning the structure of the questions.

The 2003 Sample

(Note: for a full description of sampling and response rates and results see this PDF)

In 2003, in an effort to reach a broader audience it was decided to conduct an on-line survey. The link and password to the survey was distributed on a number of on line lists (CimList for example) to at least limit the responses to members of those lists and not open to the general public. Negative, preemptive comments were almost immediately online (see Useless Online Survey of Climate Scientists). But interestingly enough, there were no overly major differences between the 1996 and 2003 results. In all there were some 500 respondents but response rates were, of course, not possible to calculate. This sampling method is known as saturation sampling. Saturation sampling attempts to survey all identifiable targets and overcomes the lack of reliable sampling frames. The low cost of internet research makes this possible.*

The 2008 Sample

Attempting to improve the survey but maintain a large sample size, three lists were employed in constructing the 2008 sample. List one included a list of authors, affiliations and email addresses drawn from climate journals with the 10 highest ISI impact ratings for the last 10 years. These are authors of climate related papers in peer reviewed climate related journals. Publication in such journals was based on the assumption that authors would be qualified to participate in a survey of climate scientists.

The second list was the list of authors who contributed to Oreskes’ conclusions concerning consensus, and the list is obtainable online at the following links:

1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003

As these authors were accepted by the broader science community as being representative of climate science we thought their inclusion would add to valid sample without controversy.

The third list was drawn from readily available email lists on institutional web sites (i.e. NCAR, MPI, AMS, etc.) Duplicates in the three lists were removed before distribution.

The combined invitation list numbered a potential 2677 respondents; defunct email addresses reduced the valid mail out to 2059. Provisions were made so that should someone submit a duplicate form the form identifier resulted in the original being over written. Consequently, for each invitation it was only possible to have one completed survey written to the data set.

The response rate for ISI authors list was approximately 27%, for Oreskes’ list, approximately 10% (some respondents took the courtesy to respond saying they were not climate scientists and were hence discounted from the possible sample), and the response rate from the Institute lists was approximately 19%, for a combined response rate of 18% (375 responses). Typical response rates for such online surveys are currently between 2% to 30%.

To conclude this section I would once more like to quote Gavin Schmidt: “

Hopefully therefore, the sampling will not be as corrupt as in the past years (response rates might still be a problem though).

The reader, of course, is free form his or her own opinion. We, however, see the sample as a valid representation of the climate science community.

Question Construction

It is somewhat difficult to decipher the message on RealClimate. On the one hand the survey is charged with having ‘too many questions’ while in the same breath with ‘aggregating multiple questions in one and not providing enough other questions …’ Our questionnaire can be viewed at on the RealClimate website where it was reproduced without notification or a request for permission to post.

In a hurried attempt to perhaps discredit the survey it seems some of the questions were misread. Gavin states:

Asking about what the ability of models is for modelling the next ten years is similarly undefined, and in fact unanswerable (since [sic] we don’t know how well they will do).

If one takes the time to look at the question (Question 16) the question reads

How would you rate the ability of global climate models to model temperature values for the next 10 years? … 50 years.

The question quite explicitly asks how well can a current model estimate the temperature 10 years into the future – isn’t that what models are intended to do? If not, how can claims be made concerning the future of the earth’s temperature?

While among many more worthy of comment, I will only address one more of Gavin’s queries. He claims question 52 is ‘a classic example of a leading question’ Question 52 reads

Some scientists present extreme accounts of catastrophic impacts related to climate change in a popular format with the claim that it is their task to alert the public. How much do you agree with this practice?

According to Gavin,

There is obviously only one sensible answer (not at all).

However, just week Roger Pielke, Jr. posted an interview with physicist Richard Muller from Berkeley endorsing exaggeration of climate science for political purposes, and the data from CliSci 2008 seems to suggest that he is not the only one to endorse such a position. The results for his question in the 2008 data are presented below.

Some scientists present extreme accounts of catastrophic impacts related to climate change in a popular format with the claim that it is their task to alert the public. How much do you agree with this practice? 1 = not at all 7 = very much

The table seems to suggest that preemptive assumptions expressed by Gavin concerning the survey might be more wishful thinking than an accurate expression. In fact, it seems that from the sample of 367 responses 53 scientists lean towards favoring the presentation of extreme scenarios to alert the public at a level of 5 or higher.

Why the paranoia about our work?

In the subsequent comments to Gavin’s posting, while the survey does get some praise, paranoid preemptive comments seem to predominate.

rpauli states

We do not need to be reminded that polling scientists will only let us know their opinions …

. . . and I have to admit that in many instances his assessment is correct. However, if one views any of the past work by the authors of the surveys of climate scientists in question, you will note that the entire question and possible response categories are presented. I agree with Murray et al (2001: 98)* that

the answers are seldom very meaningful unless you also know about the questions that elicited them.

. . . assuming of course the reader takes the time to read the question.

Dill Weed says

A well constructed survey could be very revealing otherwise it may well end up being a tool for deniers.

Is there a possibility that a poorly constructed survey could also end up a tool climate extremists?

Stephan Missal seems even more distraught:

… the flimsiness and vagueness [very explicit terms here] of the survey, combined with its flawed set of recipients [my apologies to all those who responded, but I am simply citing Mr. Missal here], makes it a poor instrument by which to make any conclusions about scientist’s opinions.

Stephan, if scientists answered the questions, what more could the answers represent than the opinions of scientists? That was the goal of the survey. As for some of the responses from the ‘flawed set of recipients’, if this represents a rather false set of conclusions one might want to rethink the climate change issue. Asking:

is climate change is happening now?
is it dangerous?
is it a result of anthropogenic causes?

produced the following sets of responses:

How convinced are you that climate change, whether natural or anthropogenic, is occurring now?
(response range: 1 = not at all; 7 = very much)

How convinced are you that climate change poses a very serious and dangerous threat to humanity? (response range: 1 = not at all, 7 = very much)

How convinced are you that most of the recent or near future climate change is, or will be, a result of anthropogenic causes? (response range: 1 = not at all, 7 = very much)

Does the fact that there is not a 100% level of agreement with your own opinion discount the results of the survey?

Oddly enough, in 2003 – the data set preemptively declared null and void – the responses were similar:

… but that is water under the bridge. It simply indicates that the ‘deniers’ are hard at it again … that is, the ‘deniers’ of the validity of surveys. When data actually supports their position, one has to wonder as to their motive and their reluctance to accept the results.

Raven says

… the surveys are simply propaganda tools…

Perhaps on should consider the user of the data produced by the survey, not the survey itself.

Stef in Canada seems to be a little ahead of himself (herself?), offering praises to the RealClimate blog:

Well-done! Thanks for letting us know what’s perking the climate science world, and also for debunking this ‘survey’

Debunking? Oh well, I guess if one is consoled by a few nameless comments on a blog … Unfortunately his (her?) advice is not too easy to understand, as Stef continues:

The best surveys are short, and come up with the answers to the questions you want.

Yeah, well, hmmm, whatever … and Stef continues:

… nonresponse rate will dominate the results.

I would assume if there are no responses there would be no results. Obviously, according to the tables above, some people responded. Why the insistence of drawing conclusion before the data is analyzed? (Likely before the data is even collected!)

I would like to respond to all comments but the silliness of them gets a bit tedious after a while. There are however a couple more worthy of note:

Stephan Linn, informs the reader that,

I could get any answer I want if I could pick the respondents

Like the response to Stef in Canada I can only suggest Yeah, well, hmmm, whatever … . Stephan Linn continues to inform us that he is,

A scientist with real credentials and peer reviewed publication.

Thanks Stephan!

Just one more, as it seems to sum up the preemptive nature of many of the comments (not all comments – some comments were very supportive).

Andre Velone says,

The results to this ‘survey’ will be biased to the deniers of climate science [please see tables above] . I hope that this post will be shared widely so that we can limit the damage of this faulty poll before it becomes used by deniers’.

Goodness gracious, Andre, at least give me time to look at the data before you deny its validity. (Do you and Stef in Canada happen to share an office?) Also, I don’t think anyone denies climate science, as far as I can determine there is a very good likelihood that it exists.

The frightening thing about the Real Climate post and the quality of debate on its site is the rush to conclusions by people insistent that any opinion other than their own is wrong. Good thing the survey asks about the acceptance of blogs as a means of communication.

Reference

*Turner WJ (1989). Small business data collection by area censusing: a field test of ‘Saturation Surveying’ methodology. Journal of Market Research Society Vol.3, No.2 April 1989.

** Murray, David, Joel Schwartz and S. Robert Lichter ‘It Ain’t Necessarily So. How Media make and Unmake the Scientific Picture of Reality. Rowman and Littlefield 2001.