Posts Tagged ‘Ryan O’Donnell’

Revkin on Steig, O’Donnell, peer review and solid scientific basics

February 12, 2011

Andy Revkin wrote a good overview of the recent kerfuffle between Eric Steig and Ryan O’Donnell. His piece is centred around contrasting the conflicting views at the edge of the scientific development with the well understood basics of scientific knowledge that makes up the big picture:

I also hope that tussles at the edges of understanding, where data are scant or uncertainty is high, don’t distract the public too much from the basics of climate science, which are boringly undisputed yet still speak of a rising risk that sorely needs addressing.

That’s a very important point to make, and I applaud Revkin for doing so. Media attention to new results (which are usually disputed to a certain extent) can sometimes lead to a skewed picture of the scientific knowledge in the field as a whole, which tend to be underreported. That’s why Revkin’s framing here is important, as it drives home the fact that a dispute at the edge of knowledge (spatial statistics as applied to Antarctic temperature trends) does not mean that the whole theory of climate change is suddenly disputed. Revkin:

Everything laid out above tends to draw attention away from the broad and deep body of work pointing to a growing and long-lasting human influence on the climate system.

Revkin does however exhibit a misunderstanding of peer review when he writes:

The exchanges between Steig and O’Donnell do raise questions about peer review, given that Steig has said he was an early anonymous reviewer (…)

This got quite a few people riled up. I wrote in to state that I think it still is a

Very good article, and good to see attention to detail not go at the cost of also providing the context of what is known.

One comment:
You say this all argument raises questions about peer review. But in fact, it is completely normal, or expected even, that authors whose paper is being critized are one of the reviewers. They are most familiar with the issues, plus it enables the editor to hear both sides.

Of course the editor needs to be aware of the position of this reviewer as the one being critiqued and weigh the review accordingly with other reviews from more disinterested parties. Revkin has since posted Louis Derry’s response, an editor of a geosciences journal:

1. Editors make final decisions. Reviewers make recommendations only.

2. It is common for a submission that critiques previous work to be sent to the author of the critiqued work for review. 2a. That emphatically does NOT mean the reviewer has veto power. It means that his/her opinion is worth having. Such a choice is usually balanced by reviewers that editors believe are reasonably independent, and the review of the critiqued is weighted accordingly. Suggestions that asking Steig to review O’Donnell was somehow unethical are utterly without support in normal scientific practice. Obviously, Steig did not have veto power over O’Donnell’s paper.

3. The fact that O”Donnell’s paper went through several rounds of review is absolutely unsurprising and unexceptional. Many papers on far less public topics do the same.

4. Some have questioned why Stieig 09 got “more” visibility than O’Donnell 10. The answer is simple. Steig had a “result,” O”Donnell had a technical criticism of methodology.

He also chimes in about the importance of the context as provided by Revkin:

Finally, Revkin’s point that the Steig vs O’Donnell debate is not unusual in the progress of science and does not have much of anything to say about the majority of the evidence is correct. Disagreement about how to model the flight of a Frisbee correctly doesn’t imply that basic aerodynamics are wrong. Disagreement about how many EOFs [empirical orthogonal functions] to use to model Antarctic [temperature] changes doesn’t imply that climate physics is wrong.

The Frisbee comment reminded me of one of my favorite sayings: Observing a bird in the sky doesn’t disprove gravity. The science may not be settled, but solid it is.

Some more things have been said about peer review by others. E.g. Andrew “Bishop Hill” Montford writes in the Hockeystick illusion, page 205 (h/t Tim Lambert):

As the CC [Climatic Change] paper was critical of his work, McIntyre was invited to be one of the peer reviewers.

 Guess we can all agree on that aspect of peer review now.

Update: John Nielsen-Gammon has some useful things to say about peer review here (on revealing the identity of reviewer), here (retelling the story and why it makes sense to have had Steig as a reviewer; quoting Steig; interesting dicussion), and here (explaining the dynamics of peer review and making the interesting suggestion of mentoring  relative outsiders navigate peer review).

%d bloggers like this: