April 2010 archive

Open Reviewing

April 29, 2010

What would open reviewing look like?

Recently, there has been a feeling that the peer review system should be revamped. We had a discussion during one of the NIPS lunchtimes about what is the future of NIPS reviewing, with many interesting suggestions. Also, several conferences have recently gone double blind. John recently blogged about compassionate reviewing.

So, following the insightful summary to the various meanings of open, including non technology ones, I thought why not open reviewing?

What is being made open?

The reviews and scores of the paper, in an open access fashion. Very much like what Yann has suggested. For true openness, the reviewer's identity should be revealed.

What legal regimes are implicated?

Since reviews today are never revealed, it seems that non even copyright is implicated. But perhaps since reviews are secret, they are covered under trade secrecy?

How does openness happen?

It can happen at an organisational level, a e.g. workshop, conference or journal can declare that all reviews are open. Or an individual can decide to make his or her reviews public. In fact, there are even two levels of public, since you can make yourself known to the authors of the paper, or you can publicly display your review for everyone to see.

We have an opportunity here to do this last idea to publicly investigate a software project. Have a look at the mloss10 submissions which are currently under review for our ICML workshop. Log in and put your reviews in the comments of the respective software projects by 6 May 2010. The program committee (whose reviews unfortunately remain secret) has exactly the same information as you do by looking at the project links.

We would like 3 scores from 1-10, with 10 being best.

1. Quality: The normal review criteria, like at JMLR
2. Potential: For very young but interesting projects
3. Interest: How interesting the software is for the ML community

Deadline Extension and Final Call for Contributions ICML'10 Workshop

April 12, 2010

To accomodate researchers waiting for decisions on their ICML papers (due April 16) before committing to travel to Haifa, the submission deadline for the Machine Learning Open Source Software (MLOSS) 2010 workshop has been extended to April 23. As a result of this, we have also pushed back the acceptance notification to May 8. The workshop will take place at ICML 2010, Haifa, Israel, 25th of June, 2010.

Nevertheless, the deadline for the submissions is approaching quickly. We accept all kinds of machine learning (related) software submissions for the workshop. If accepted, you will be given a chance to present your software at the workshop, which is a great opportunity to make your piece of software more known to the machine learning community and to receive valuable feedback.

Detailed submission instructions are available at http://mloss.org/workshop/icml10/. We are looking forward to your contributions.