July 2009 archive

A Machine Learning Forum

July 21, 2009

Yoav Freund has started a new discussion forum. It already has quite a few contributors, many whose names would be familiar to machine learners. At the moment, it seems to be mostly a collection of tutorials and introductory articles, but in principle it could become a good place for open discussions.

Chris Drummond suggested that what we really need is a place for open debate. Perhaps a letters section in JMLR. Would the above forum suffice? Or do we need some sort of quality control of the published ideas (hence increasing reviewer load)?

Getting rid of Spam

July 13, 2009

You might have noticed that mloss.org received increasing amounts of spam starting from June. Well actually not spam but scam (garbage trying to poison spam filters - random sentences very hard to filter out in an automated way). For that reason I have to disable comments for anonymous, i.e., users who are not logged in. Sorry for the inconvenience.

Replicability is not Reproducibility: Nor is it Good Science

July 13, 2009

I had promised Soeren that I would post a short version of my argument so we could debate it in this forum. As Cheng Soon kindly points out there is a longer version available.

One compelling argument for repositories such as mloss is reproducibility. Reproducibility of experimental results is seen as a hallmark of science. By collecting all the artifacts used in the production of the experimental results reported in a paper would, it is claimed, guarantee reproducibility. Although not explicitly stated, the subtext is that if we have any pretensions of being scientists then we have little choice but to do this.

My counter argument is that this view is based on a misunderstanding of reproducibility in science. What mloss will allow people to do is replicate experiments, but this is not reproducibility. Reproducibility requires changes; replicability avoids them. Reproducibility's power comes from the differences between an original experiment and its reproduction. The greater the difference the greater the power. One important role of an experiment is to support a scientific hypothesis. The greater the difference of any subsequent experiment the more additional support garnered. Simply replicating an experiment would add nothing, except perhaps to confirm that the original was carried out as reported. To me, this is more of a policing exercise than a scientific one, and therefore, I would claim, of much reduced merit.

GCC + Machine Learning

July 5, 2009

I found a cool project recently which applies machine learning for something that affects most of us who write software. Milepost is a project that uses statistical machine learning for optimizing gcc. They point to cTuning.org for further development. Just to quote that they hope to do with milepost gcc:

"Next, we plan to use MILEPOST/cTuning technology to enable realistic adaptive parallelization, data partitioning and scheduling for heterogeneous multi-core systems using statistical and machine learning techniques."

There is a lot of infrastructure that needs to be built before coming to the machine learning. In the end, the machine learning question can be stated as follows:

Given M training programs, represented by feature vectors t1,...,tM, the task is to find the best optimization (e.g. compiler flags) for a new program t. In the standard supervised setting way, they collect training data for each program ti consisting of optimization (x) and run time (y) pairs. Then the machine learning question boils down to finding the parameters theta of a distribution over good solutions, q(x|t,theta). I.e. the right compiler settings (x) for a given program (represented by features t).

However, it seems that they use uniform sampling to search q(x|t,theta) for good solutions, and once they have these islands of good solutions they use 1-nearest neighbour for prediction. There seems to be a lot of scope for improvement on the machine learning side.