Thursday, May 16, 2013

The Past is Prologue

Cook et al., searched the ISI web of science between 1991 and 2011 for articles which were returned following a keyword search on "global warming" and "global climate change".  A team of reviewers then categorized the ABSTRACTS.  Back in 2008, Brian organized a similar exercise for a smaller number of abstracts, about 650 published between 2002 and 2006 which will be designated below as the prequel.  A team of six reviewers then categorized the ABSTRACTS with two reviewers looking at each abstract.

As a first step it is useful to compare the categories which were independently settled on by both teams.  Until about a week ago, neither team was aware of the others work, so these were indeed independent exercises..

Prequel
Explicit Endorsement of AGW
Explicit Endorsement of AGW
Implicit Endorsement
Implicit Endorsement
No Position
Discussion
Implicit Rejection
Methods
Explicit Rejection
Paleoclimate

Unrelated

Undecided

Implicit Rejection

Explicit Rejection

The primary difference is that the Prequel Team had more categories in the middle to deal with the predictable muddle.

An important point that has been missed is that MOST, essentially the vast majority of the papers, tossed up by such a search are biologically, ecologically or economically oriented.  Relatively few are what the Bunnies would call WGI material.  Most are WGII and WGIII.  Go ahead try it in Google Scholar.  Such papers, like WGII and WGII one way or another take the conclusions of WGI (that there is human caused climate change) as their starting point and we had to figure out how to deal with that

The prequel of the prequel was a small training set selected from the ~650 from which rating instructions were developed and then each member of the team rated about 200 abstracts.  The results were


Prequel
#

Explicit Endorsement of AGW
244
21%
Implicit Endorsement
407
34%
Discussion
222
19%
Methods
78
  7%
Paleoclimate
107
  9%
Unrelated
43
  4%
Undecided
33
  3%
Implicit Rejection
21
  2%
Explicit Rejection
28
  2%



Cross-Tabs tomorrow, but in almost all the cases the two reviewers were close to each other lumping the five neutral ratings as one.  There were only 13 problem papers where there was significant disagreement (support/reject) between the two reviewers, at which point everyone ran out of steam before trying to to reconcile them by actually looking at the papers.

Why bring this up now.  Well there have been a bunch of surveys, this being an unpublished one, and they all are within a stone's throw of each other, the "doubting" literature is really thin.

Congratulations to John, Dana, and the rest of the authors and helpers from Skeptical Science.  The amount of work they did is astounding.  Eli knows.  Brian is still in the recovery room.

6 comments:

John Cook said...

Eli, since completing the Consensus Project, I've learnt of two efforts to measure the consensus in peer-reviewed papers - both cases rating less than 1000 abstracts and both cases failing to complete and publish the project. If I'd been aware of this at the start of our project, I probably wouldn't have opted for a sample size of 12,000+ abstracts!

Susan Anderson said...

Congratulations to John Cook on getting the attention of the big Kahuna (and wish he'd get a spine and stop thinking like a lawyer about science on climate):

http://www.smh.com.au/environment/climate-change/obama-gives-aussie-researcher-31541507-reasons-to-celebrate-20130517-2jqrh.html
(h/t Tenney Naumer)

"Obama gives Aussie researcher 31,541,507 reasons to celebrate

Peter Hannam, Carbon economy editor, The Sydney Morning Herald, May 17, 2013

Ted Kirkpatrick said...

Eli, typo alert: "The primary difference is that the Prequel team had FEWER categories in the middle ..."

coby said...

I always wondered what happened, but since the effort seemed to peter out right along with my time for it I never really inquired...

willard said...

Are you an alarmist?

Are you a wacked out sceptic?

If you answer no to these two questions, chances are you are lukewarm.

Why, you may ask?

Because the Pope of the Lukewarm church proclaimed so:

> If I choose to divide the world into 3 classes: wacked out alarmists. Wacked out skeptics; and the sane middle ground, you dont get to challenge my classification. You simply dont get to challenge it. And in the end you will see that 97% of people are
in the middle, as I define it.

http://rankexploits.com/musings/2013/on-the-consensus/#comment-113304

In his infinite wisdom, the Pope of the Lukewarm Church made it clear that you don't get to challenge that.

Welcome aboard!

EliRabett said...

In thinking about it, Eli thought that the five categories

Discussion
Methods
Paleoclimate
Unrelated
Undecided

were all in the middle between reject and accept