[UPDATE: Prof. O'Hagan explains in the comments that his response reflects his personal views and not those of RMS, so I have altered the title.]

Last week I argued that the RMS expert elicitation of expert views on hurricane landfalls over the next five years gave a result no different than if a bunch of monkeys had engaged in the elicitation. Professor Tony O’Hagan who conducted the elicitation on behalf of RMS responds in the comments to that thread. Below I have reproduced his comments along with my rejoinders provided in bold:

I am the statistician who conducted the expert elicitation that Dr Pielke derides. I feel that I must answer his unbalanced criticism of the procedure that I adopted in collaboration with RMS. Like Dr Pielke, I was engaged by RMS as an expert to help them with the assessment of hurricane risks. My skills are in the area of probability and statistics, but in particular I have expertise in the process of elicitation of expert judgements. I am frequently dismayed by the way that some scientists seem unprepared to acknowledge the expertise of specialists in other fields from their own, and seem willing to speak out on topics for which they themselves have no specific training. During the elicitation exercise it was essential for me to trust the undoubted expertise that he and the other participants had in the science of hurricanes, and I wish that he had the courtesy to trust mine.

PIELKE RESPONSE: Prof. O’Hagan is apparently unaware that I am trained in the social sciences, with social science methodology as one of my major fields.

Let me now address Dr Pielke’s specific criticisms.

First, he says that the results obtained were indistinguishable from the results of randomly allocating weights between the various models, and he implies that this is inevitable. The latter implication is completely unjustified. I was not involved in the 2006 elicitation which Dr Pielke uses for his numerical illustration, but I can comment on the two most recent exercises. The experts were given freedom to allocate weights, and did so individually in quite non-random ways. In aggregate, they did not weight the models at all equally. The fact that the result came out in the middle of the range of separate model predictions in 2006 was therefore far from inevitable.

PIELKE RESPONSE: I am happy to see Prof. O’Hagan acknowledge that the results were indistinguishable from a random allocation. On this point we agree. Prof. O’Hagan can further clarify the situation by releasing the values for the five-year predictions from the 39 models used in the 2008 elicitation. I ask him to submit these in the comments to this thread.

The elicitation exercise was designed to elicit the views of a range of experts. They were encouraged to share their views but to make their own judgements of weights. Dr Pielke says that the more experts we have, the more likely it is that the elicited average will come out in the middle, which is again fallacious. The result depends on the prevailing opinions in the community of experts from whom the participants were drawn. The experts who took part were not chosen by me or by RMS but by another expert panel. If, from amongst the models that RMS proposed, all the ones which would give high hurricane landfalling rates were rejected (and so given very low weights) by the experts, then the result would have ended up below the centre of the range of model predictions. The fact that it comes somewhere in the middle is suggestive, if it suggests anything at all, of RMS having done a good job in proposing models that reflected the range of scientific opinion in the field.

PIELKE RESPONSE:Again, I am happy to see Prof. O’Hagan acknowledge that the experts did little more than confirm the distribution of models presented by RMS. I will reiterate that a group of experts with a wide range of views, such as found in the tropical cyclone community, will inevitably provide a result indistinguishable from a set of random views, just as a panel of monkeys allocating random weights would have done, as I argued in my earlier post. This point is simply a logical one. If the community had a consensus, presumably such an elicitation would be unnecessary.

I think the above also answers Dr Pielke’s criticism of RMS’s potential conflict of interest. I agree that this potential is real. RMS is a commercial organisation and their clients are hugely money-focused. Nevertheless, as I have explained, the outcome of the elicitation exercise is driven by the judgements of the hurricane experts like Dr Pielke. Any attempt by RMS to bias the outcome by proposing biased models should fail if the experts are doing their job. If Dr Pielke is convinced, as he appears to be, that no model can improve on using the long-term average strike rate, then he could have allocated all of his weight to this model. That he did not do so is not the fault of RMS or of me.

PIELKE RESPONSE: It is telling Prof. O’Hagan sees fit to attempt to reveal publicly my individual allocations in the exercise after the participants in the elicitation were assured by him and RMS that any individual information would remain confidential. I am sure that there are other “confidential” details about the elicitation that many people would be interested to hear about. I remain perfectly comfortable with my allocation in the process despite the fact that I could have been replaced with a monkey to no real effect on the outcome. My views on one to five year predictions are expressed in a paper (currently under review) that I am willing to share with anyone interested (pielke@colorado.edu).

This brings me back to the question of expertise. The elicitation was carefully designed to use to the full the expertise of the participants. We did not ask them to predict hurricane landfalling, which is in part a statistical exercise. What we asked them to do was to use their scientific skill and judgement to say which models were best founded in science, and so would give predictions that were most plausible to the scientific community. I believe that this shows full appreciation by RMS and myself of the expertise of Dr Pielke and his colleagues. For myself, the expertise that Dr Pielke seems to discount completely is based on familiarity with the findings of a huge and diverse literature, on practical experience eliciting judgements from experts in various fields, and on working with other experts in elicitation. In particular, I have collaborated extensively with psychologists and other social scientists. I don’t know how much Dr Pielke knows of such things, but to complain that what I do is “plain old bad social science” is an insult that I refute utterly.

PIELKE RESPONSE: I would agree that Prof. O’Hagan does not know how much I am aware of such things.

Dr Pielke is no doubt highly-respected in his field, but should stick to what he knows best instead of casting unfounded slurs on the work of experts in other fields.

PIELKE RESPONSE: Academics sometimes like to conflate a professional critique with a personal “slur,” perhaps to change the subject. I have the highest respect for RMS as the leading catastrophe modeling firm with an important role in the industry. It is the importance of RMS to business and policy that merits the close attention to what they are doing.

In this case, I judge the elicitation methodology to be significantly flawed in important respects. This perspective is no “slur,” just reality. Prof. O’Hagan can help to further clarify the situation by focusing on the critique rather than expressions of outrage. He might start by releasing the results from the 39 models used in the 2008 elicitation. I ask that he publish these in the comments to this thread.