Anonymous or open review

I believe technology can be used to create a review system that resolves many of the tensions between accountability and anonymity.

Anonymous or open review

Thu 28 July 2011

There’s an ongoing debate in academia on the merits of the traditional anonymous peer review process. After submitting a paper for anonymous peer review only the review itself is made public; the identity of the reviewer is protected. The alternative proposed is open peer review where the names of reviewers are published.

While anonymous peer review provides protection from backlash after a harsh review, it’s criticised for encouraging a closed shop where papers challenging conventional thinking are less likely to be well received. Open review, on the other hand, is regarded as making reviewers too cautious. The limited hard data available indicates that academics are less willing to be reviewers under the open system.

My interest in this debate centres on the role technology might play in future review processes.

For a recent client project we instituted open group peer review of research funding applications. By “open group” I mean that the reviewers are preselected and aware of the group composition. There’s a web interface which allows applications to be read, reviews to be posted and – because the aim was to accept or reject the proposals – allows the reviewers to post a score for each application. Reviews and scores are visible to all reviewers. The reviewers have been slow to adopt the new system. What we don’t presently understand is whether this reluctance stems from having to put their reviews on record, or from the technology itself.

The primary concern in any peer review process should be rigour. Reviews must be informed and thorough and, so far as humanly possible, unaffected by personal bias or professional rivalry. It’s clear that anonymous peer review is imperfect, but it’s equally clear that open peer review is not a perfect alternative.

Can technology enable a hybrid peer review system which offers the best of both approaches? Perhaps.

Designing such a system isn’t trivial and, as my own experience (see right panel) indicates, its ultimate success will largely hinge on the level of involvement. Submitters and reviewers need to be eager participants who prefer using the system to any of the alternatives. To design the irresistible system we would need to learn some lessons from social networking models and internet forums.

  • Optional anonymity?
  • Benefits from disclosure of one’s own identity?
  • A weighted peer rating process for reviewers and/or reviews?
  • An initial moderation period for new participants?
  • Configurable visibility for one’s own identity and/or reviews?

No, I’m not proposing a design – that’s beyond the scope of this piece. But I believe technology can be used to create a review system that resolves many of the tensions between accountability and anonymity. And I think the design of such a system would be a fascinating project.

But when the design’s complete … how would we review it?

What do you think?

Comments are aggressively moderated. Your best chance is reasoned disagreement.

*