The mloss.org community blog

Which programming language on mloss.org?

Cheng Soon Ong — Mon, 22 Dec 2008 17:02:57 -0000

The winner is suprisingly (for me at least) C++. And here we all thought that machine learners could only program in matlab. Speaking of matlab, why are there so many projects with matlab sources, but so few with octave?

The runner up, R, is mostly automatically obtained from the CRAN servers, so assuming that this is somehow at steady state and assuming that all languages have equal numbers of supporters, we should expect that all the major languages have more or less 40 projects each. Rounding up the top are C, Python and Java.

Here's the numbers for the site:

C:37
C++:49
D:1
Erlang:1
Java:22
lisp:2
Lua:1
Matlab:37
Octave:4
Perl:4
Python:34
R:40
Ruby:2

MLOSS Workshop Videos online

Mikio Braun — Mon, 22 Dec 2008 15:26:30 -0000

The videos for our MLOSS workshop at Whistler are online at videolectures.net. Also, the discussions were recorded on video. At around 19:50 in the Video on Reproducibility, you can see me trying very hard to keep focused while not staring at the camera.

Happy Holidays, everyone!

On our NIPS workshop

Mikio Braun — Fri, 19 Dec 2008 17:35:31 -0000

On December 12, our third workshop on machine learning open source software was held in Whistler, BC, Canada. It featured two invited speakers, a host of new and exciting software projects on machine learning, and two interesting discussions where we tried to initiate new developments.

We were very glad to get two speakers from very prestigious and successful projects: John W. Eaton from octave, a matlab clone, and John D. Hunter from matplotlib, a plotting library for matlab.

John W. Eaton gave valuable insights into his experiences in running an open source project. Started in 1989 as companion software to a book on chemical reactions, its main intention was to give students something which is more accessible than Fortran. Only afterwards were people realizing that octave was very close to matlab, and over the years people were requesting better and better compatibility to matlab. The last major release, version 3, has brought even fuller compatibility with the support of sparse matrices, and a complete overhaul of the plotting functionalities. Still, octave is searching for help, in particular in the areas of documentation, mailing list maintenance, and packaging. So if you're interested, drop John a line.

Matplotlib by John D. Hunter also started as a private project. John worked on epilepsy research in neurophysiology and initially wrote what would become matplotlib to display brain waves together with related data. At some point, matplotlib has started to become so big that it practially required all of his time. By now, John is working in the finance industry but has an agreement with his employer to work on matplotlib a certain fraction of his time. We have also learned that matplotlib contains a full re-implementation of the TeX algorithms by Donald Knuth for rendering annotations in the plot.

Both speakers stressed the importance of being resilient and pointed out that they both had to go through some time (might even be years) before a project really takes off. Both also shared their insights on how difficult it can be to deal with users. On the one hand, you have to be reliable to build up trust in your project, on the other hand, there are always some users who expect full support basically for free and are unwilling to contribute.

Besides those two invited talks, we again had a number of interesting projects. The submissions this year could roughly be classified into full frameworks, projects which focus on a special type of application or algorithm, and infrastructure.

We had four different projects which are providing a full-blown environment for doing machine learning and statistical data analysis. The first talk was Torch, a full blown matlab-replacement written in a combination of Lua and C++. Torch is optimized for efficiency and large scale learning and comes with its own matrix classes (called tensors), and plotting routines. Shark is a similarly feature rich framework written in C++. For users of R, there is [kernlab][ker] which focuses on kernel methods. Finally, python was represented by mlpy, and mdp, which sported an innovative module architecture which allows to plug together data processing modules. It was very interesting to see that there exist so many different projects which have such a broad scope. It was also quiet interesting to learn that these projects weren't so much aware of one another.

Projects which were more focused on a smaller scale included Nieme, which contains algorithms for energy-based learning, libDAI, a library of inference algorithms for graphical models with discrete state spaces, and Model Monitor, a tool for assessing the amount of distribution shift in the data and sensitivity of algorithms under distribution shift. The BCPy project again provides a python layer over the BCI2000 system and allows to work with the later in a much more flexible manner.

Finally, we had projects which dealt with different aspects of infrastructure. The RL Glue project provides a general framework to connect environments and learners in a reinforcement learning framework. This project has been highly successful, and is the standard platform for a number of challenges in this area. Disco implements the map-reduce framework for distributed clustering in a particularly elegant manner for python users, based on a core is written in Erlang. The Experiment Databases for Machine learning and BenchMarking Via Weka projects address the issue of benchmarking machine learning algorithms in an automatic and reproducible way and providing a database to describe models and experimental results.

In summary, it seems that researchers are quite active in providing feature-rich high-quality open source software on machine learning. The large number of 23 submissions to this workshop also provides evidence for that. At the same time, it seems that most projects are still oblivious of each other. In particular, when it comes to interoperability, it seems that there is still a lot missing, making it hard to combine algorithms written in different languages, or code developed with respect to different frameworks.

Therefore, one of the discussions was focused on the question of interoperability. As a starting point, we proposed the ARFF file format as a common file format for exchanging data. Such a file format could serve as an important first step. Leaving more complex solutions like remote method invocation or CORBA aside, a common data format is really the simplest way to exchange data between two pieces of code which might be written in different languages or run on different platforms. As we expected, the discussion was quite lively, as the number of possible data formats is large, and the different features you could want are not always compatible. But I think what we achieved was to raise awareness for the need of interoperability. Hopefully, people will start to think about how their code could interact with other code, and standards will emerge over time.

The other discussion addressed an even more difficult question, namely that of reproducibility. How can we ensure that somebody else can reproduce the experimental results from a machine learning paper? An interesting suggestion was to require that the software producing the results is provided on a bootable live CD like a Ubuntu install CD to really make sure that the environment in which the experiments were done can be set up easily. The question was also whether you want to be able to reproduce the results at publication time, or even after ten years. Again, there is the problem of how to describe and store results in a database. Here also, we did not arrive at a conclusion, but the overall awareness could be raised hopefully.

Overall, I think the workshop was very successful and interesting. Room for improvement is always there, of course. For example, we should make sure not to forget to schedule coffee breaks next time. Also, I think we should put more emphasis on the community building aspect and less on individual projects. In 2006, the topic was so new that people didn't know what kinds of projects were out there, but now, also due to this website, the existence of open source software for machine learning is much more known. So giving projects a platform to advertise their software is certainly an important part, but thinking about what the next step is and talking about how to integrate what we already have is something I would put more emphasis on next time.

Again I (and Soeren and Cheng as well) would like to thank everybody who contributed to this workshop, and of course also the Pascal2 framework for their financial support.

Help needed

Cheng Soon Ong — Mon, 08 Dec 2008 18:27:34 -0000

NIPS is a lot like Christmas. You travel to some place once a year, meet people (some you like, others you don't), and eat a lot.

To the point of this entry. If there are any budding Django programmers out there who would like to help out with the development of mloss.org, please come and talk to us. We will have a t-shirt handing out table again at NIPS, so please come by and have a chat with myself, Soeren or Mikio.

MLOSS progress updates for November 2008

Soeren Sonnenburg — Thu, 27 Nov 2008 16:24:17 -0000

Two months passed since the last statistics update, so lets see if we are progressing:

As of today mloss.org has

158 software projects based on
19 programming languages,
302 authors (including software co-authors),
284 registered users,
63 comments (including spam :),
109 forum posts,
28 blog entries,
51 software ratings,
31525 software statistics objects,
143 software subscriptions or bookmarks.

And happy birthday mloss.org - the site is live for 1 year and 1.5 months now and since we became a recent target of spammers it might show that mloss.org is not that unimportant anymore. This is also documented by a traffic growth from around 300 visits per week (February 2008) to more than 1000 per week (November 2008).

And congratulations Peter Gehler, author of the most successful software project: MPIKmeans (accessed more than 6000 times).

Finally JMLR-MLOSS received

20 submissions until now,
5 resubmissions;
3 are already accepted and published,
1 is pending publication

since its announcement in summer 2007.

One may conclude - there is visible progress. However, as already pointed out in several previous blog posts - we merely see several isolated mloss projects that don't at all inter-operate with each other. And it is clear that this trend needs to be stopped, but how could we support the next steps? In case you have some bright ideas either talk to us at NIPS*08 (possibly even attend the workshop and present your ideas in the discussion) or leave a comment...

Open Source is not Interoperability

Cheng Soon Ong — Wed, 26 Nov 2008 14:03:41 -0000

Trying to prepare some thoughts about interoperability to be discussed at the NIPS workshop, I came across a bunch of websites roughly in the following order:

a rather negative article about the state of open source software and how they interoperate at eweek.com.
a very positive blog at ugotrade which talks about OpenSim and how it will be the next hot thing.
a post about why open source and interoperability are really two different things.

Quoting the third author:

Interop is not open source.
Interop does not require open source implementations
Open source does not guarantee Interop

While one thinks that somehow it is natural for open source developers to make use of other bits of (open source) software, it usually doesn't happen. For me, interoperability can occur in two ways: the first being having a common set of protocols (as argued for by the third post above), and/or the second which is integrating another software library or method. In some sense, the "integration" idea also requires a set of protocols or APIs. It may be that I'm just being pedantic about trying to semantically differentiate between protocols and APIs. But the main idea remains: We need software that talks to other bits of software.

However, if both pieces of software are open source, we can do more than just have software that talks to other bits of software (which is why OpenSim is raising so much interest). In the process of having to push together two software projects, we may be able to come up with better interfaces between them. This is especially true in the research area (which in some sense practices carpentry) where it is not that clear from the start how programs should interact. For supervised machine learning, datasets are a good place to start. It seems "obvious" that this is one place where different machine learning algorithms can interface with each other. Even in this "simple" interface, there is a multitude of data formats and standards. Another quite fruitful area is in convex optimization, where there are several projects (even here on mloss.org) which easily link to different back ends, or several solvers which are used by various front ends. Interestingly, here the interfaces are actually dictated by the mathematics, and the software implementations are just mirroring these forms. I think it is within our reach to have these kinds of interoperability for many other areas of machine learning.

As for the long term goal of software systems being well integrated in the application specific fashion, I think we still have a way to go yet...

mloss08 Program

Cheng Soon Ong — Thu, 06 Nov 2008 15:01:13 -0000

Just in case you haven't checked our workshop page recently, we have finalised our program. We had a surprisingly large number of submissions, ranging from quite mature projects to small radical ideas. In the end, we decided that we should try to squeeze in as many projects as possible, and at the same time try to keep some diversity in the program; i.e. we didn't want to have all slots taken up by large mature machine learning frameworks.

Our theme this year is "interoperability, interoperability, interoperability". The dream is to have some way for machine learning software to talk to each other. We are still a long way from being able to plug and play different tools for machine learning, and we hope to make a start by discussing this at the workshop. Of course, machine learning research is not only about software, but it is also about the data. Our afternoon discussion session will be about "UCI 2.0", and how we should go about it. There was a recent editorial in Nature Cell Biology about the need for standardizing bioinformatics data, and this blog post highlights three properties of scientific data.

Hope to see you at NIPS!

Reviewing software

Cheng Soon Ong — Mon, 13 Oct 2008 09:40:59 -0000

The review process for the current NIPS workshop mloss08 is now underway. There are a couple of interesting thoughts that I had while discussing this process with Soeren and Mikio, as well as some of the program committee. The two issues are:

Who should review a project?
What are the review criteria?

Reviewer Choice

Unlike standard machine learning projects, choosing a reviewer for a mloss project has to be comfortable with three different aspects of the system, namely:

The machine learning problem (e.g. Graphical models, kernel methods, or reinforcement learning)
The programming language, or at least the paradigm (e.g. object oriented programming)
The operating environment (which may be a particular species of make on a version of Linux)

There is also projects about a particular application area of machine learning, such as brain-computer interfaces which put an additional requirement on the understanding of the reviewer.

However, if one looks at the set of people who satisfy all those criteria for a particular project, one usually ends up with only a handful of potential researchers, most of which would have a conflict of interest with the submitted project. So, often I would choose a reviewer who is an expert in one of the three areas and hope that he or she would be able to figure out the rest. Is there a better solution?

Review Criteria

The JMLR review criteria are:

The quality of the four page description.
The novelty and breadth of the contribution.
The clarity of design.
The freedom of the code (lack of dependence on proprietary software).
The breadth of platforms it can be used on (should include an open-source operating system).
The quality of the user documentation (should enable new users to quickly apply the software to other problems, including a tutorial and several non-trivial examples of how the software can be used).
The quality of the developer documentation (should enable easy modification and extension of the software, provide an API reference, provide unit testing routines).
The quality of comparison to previous (if any) related implementations, w.r.t. run-time, memory requirements, features, to explain that significant progress has been made.

This year's workshop has the theme of interoperability and coorperation. Therefore it is also a review criteria. The important question is how to weight the different aspects? The answer is not at all clear. There is a basic level of adherence which is necessary for each of the criteria, above which is it difficult to trade off the different aspects quantitatively. For example does very good user documentation excuse very poor code design? Does being able to run on many different operating systems excuse very poor run time memory and computational performance?

Put your comments below or come to this year's workshop and discuss this!

GNU Octave on Free Software Foundations High Priority List

Soeren Sonnenburg — Mon, 06 Oct 2008 16:46:04 -0000

The Free Software Foundation (FSF) maintains a high priority list of software projects and can be found here.

Quoting the FSF:

The FSF high-priority projects list serves to foster the development of projects that are important for increasing the adoption and use of free software and free software operating systems. [...] Some of the most important projects on our list are replacement projects. These projects are important because they address areas where users are continually being seduced into using non-free software by the lack of an adequate free replacement.

With rank eight among the top ten prioritized software projects is GNU Octave --- a free software Matlab replacement.

As this is very relevant to our community that is strongly dominated by Matlab, I would like to encourage everyone to try out octave 3.0. If you tried octave 2.x or any earlier version at some point, it really matured a lot. It supports all the data types like cell arrays, dense or sparse arrays you know from matlab and yes it has all these plotting functions like plot, surf3d etc too. And if you ever tried to extend matlab using C code, support is really much better from the octave side not to mention the killer feature: Octave is fully supported by swig! Still not convinced? We will have John W. Eaton to introduce octave to us at the NIPS'08 MLOSS Workshop. So what are you waiting for, give octave a try and see how you can help!

Differences between paid and volunteer FOSS contributors

Soeren Sonnenburg — Fri, 03 Oct 2008 08:53:25 -0000

I just stumbled across a very interesting article titled Differences between paid and volunteer FOSS contributors that I am going to almost fully quote below. The original article was written by Martin Michlmayr and can be found here. Almost full quote follows:

There's a lot of debate these days about the impact of the increasing number of paid developers in FOSS communities that started as volunteer efforts and still have significant numbers of volunteers. Evangelia Berdou's PhD thesis "Managing the Bazaar: Commercialization and peripheral participation in mature, community-led Free/Open source software projects" contains a contains a wealth of information and insights about this topic.

Berdou conducted interviews with members of the GNOME and KDE projects. She found that paid developers are often identified with the core developer group which is responsible for key infrastructure and often make a large number of commits. Furthermore, she suggested that the groups may have different priorities: "whereas [paid] developers focus on technical excellence, peripheral contributors are more interested in access and practical use".

Based on these interviews, she formulated the following hypotheses which she subsequently analyzed in more detail:

Paid developers are more likely to contribute to critical parts of the code base.
Paid developers are more likely to maintain critical parts of the code base.
Volunteer contributors are more likely to participate in aspects of the project that are geared towards the end-user.
Programmers and peripheral contributors are not likely to participate equally in major community events.

Berdou found all hypotheses to be true for GNOME but only hypothesis two and four were confirmed for KDE.

In the case of GNOME, Berdou found that hired developers contribute to the most critical parts of the project, that they maintained most modules in core areas and that they maintained a larger number modules than volunteers. Two important differences were found in KDE: paid developers attend more conferences and they maintain more modules.

Berdou's research contains a number of important insights:

Corporate contributions are important because paid developers contribute a lot of changes, and they maintain core modules and code.
While it's clear that the involvement of paid contributors is influenced by the strategy of their company, Berdou wonders whether another reason why they often contribute to core code is because they "develop their technical skills and their understanding of the code base to a greater extent than volunteers who usually contribute in their free time". It's therefore important that projects provide good documentation and other help so volunteers can get up to speed quickly.
Since many volunteers cannot afford to attend community events, projects should provide travel funds. This is something I see more and more: for example, Debian funds some developers to attend Debian conference and the Linux Foundation has a grant program to allow developers to attend events.
Paid developers often maintain modules they are not paid to directly contribute to. A reason for this is that they continue to maintain modules in their spare time when their company tells them to work on other parts of the code.

The rest of the article can be found here.