RSS Feed - BlogOpen Reviewing
Posted by Cheng Soon Ong on April 29, 2010
What would open reviewing look like?
Recently, there has been a feeling that the peer review system should be revamped. We had a discussion during one of the NIPS lunchtimes about what is the future of NIPS reviewing, with many interesting suggestions. Also, several conferences have recently gone double blind. John recently blogged about compassionate reviewing.
So, following the insightful summary to the various meanings of open, including non technology ones, I thought why not open reviewing?
What is being made open?
The reviews and scores of the paper, in an open access fashion. Very much like what Yann has suggested. For true openness, the reviewer's identity should be revealed.
What legal regimes are implicated?
Since reviews today are never revealed, it seems that non even copyright is implicated. But perhaps since reviews are secret, they are covered under trade secrecy?
How does openness happen?
It can happen at an organisational level, a e.g. workshop, conference or journal can declare that all reviews are open. Or an individual can decide to make his or her reviews public. In fact, there are even two levels of public, since you can make yourself known to the authors of the paper, or you can publicly display your review for everyone to see.
We have an opportunity here to do this last idea to publicly investigate a software project. Have a look at the mloss10 submissions which are currently under review for our ICML workshop. Log in and put your reviews in the comments of the respective software projects by 6 May 2010. The program committee (whose reviews unfortunately remain secret) has exactly the same information as you do by looking at the project links.
We would like 3 scores from 1-10, with 10 being best.
- Quality: The normal review criteria, like at JMLR
- Potential: For very young but interesting projects
- Interest: How interesting the software is for the ML community
Comments
-
- Cheng Soon Ong (on June 22, 2010, 21:58:40)
Someone asked me recently whether it was legal to put the reviews received on their paper online. My short (and disappointing) answer was "I don't know".
This is the flip side of open reviewing. Are authors allowed to also put a copy of the reviews they have received online? Perhaps they are particularly proud of a positive comment from an anonymous reviewer?
I feel that putting the reviews (anonymous of course) online would already go halfway to the "reputation" problem.
-
- Cheng Soon Ong (on June 24, 2010, 17:57:41)
The neuroscience community has put some effort into a "journal club".
http://thirdreviewer.com/
Leave a comment
You must be logged in to post comments.
Asking reviewers to perform open reviews is asking a lot. Most significantly, it is asking them to put their reputations on the line. Further, the reviews themselves would be less meaningful if people who cared about their reputations would shy away from participating.
To mitigate this, I propose that it is necessary to have dynamic reviews that reviewers can edit anytime. Of course, this would require many changes to ripple through the system. For example, it would no longer be reasonable to accept or reject a paper if the reviews are dynamic. Thus, journals would immediately publish everything that is submitted, but give each paper a ranking that changes as the reviews change. A high ranking is the new "acceptance", and a low ranking is the new "rejection". This would also help to solve certain other problems, such as long time-to-publication, and reviewers accepting papers that they cannot understand.
Journals should also recognize the effort of reviewing as a positive contribution to the scientific community by giving awards and titles that reviewers can add to their resumes. This, of course, would require some form of meta-review process.
An author should be able to pull a paper if he/she is no longer happy with its current ranking at a particular journal, and send it to another journal. This way, authors are also safe from getting burned by the dynamic review system.
Perhaps after several years, citations will gradually weigh more heavily on the rankings of papers than review scores. Thus, young papers are ranked by the potential that reviewers predict, and older papers are ranked by the impact they have demonstrated.