RL Glue and Codecs -- Glue 3.x and Codecs

Brian Tanner, Adam White, Richard S. Sutton — Mon, 12 Oct 2009 07:50:03 -0000

Please go to http://glue.rl-community.org for information on how to cite RL-Glue if you are using in your own research and publications.

RL-Glue allows agents, environments, and experiments written in Java, C/C++, Matlab, Python, and Lisp to inter operate, accelerating research by promoting software re-use in the community.

Note: This is a release candidate, not a final release. We are actively soliciting feedback from the community about any problems with the software or documentation that can be improved before a final release in January 31 2009.

Update posted: Dec 10/2008. Some updates to the protocol, notably removing the seed/key methods. This should be the last release candidate before the final release on January 31 2009.

Update posted: Oct 11/2008, Big change for main RL-Glue and C-Codec to use const-pointers instead of structs by value in parameters and return types. Breaks backward compatibility. See the tech manual in docs.

Update posted: Oct 8/2008, fixed memory leak in RL-Glue, fixed skeleton experiment build on Linux, updated some Cygwin compatibility stuff.

Overview

Inspired by related psychological theory, in computer science, reinforcement learning is a sub-area of machine learning concerned with how an agent ought to take actions in an environment so as to maximize some notion of long-term reward. Reinforcement learning algorithms attempt to find a policy that maps states of the world to the actions the agent ought to take in those states... -- Wikipedia Reinforcement Learning Article

RL-Glue is a set of common guidelines for the reinforcement learning community to follow to allow us to share and compare agents and environments with greater ease. The software implementation of RL-Glue is the reusable glue interface to connect the basic parts of a learning experiment.

RL-Glue supports interaction between agents, environments, and experiment programs in two different modes. In direct-compile mode, all three modules are written in C/C++ and compiled together into a single executable program.

In the more flexible socket mode: agents, environments, and experiments use inter-process communication through sockets, either locally on one computer or over the network or Internet. In socket mode, agents, environments, and experiments written in a variety of languages can interact with each other transparently. The language-specific software that allows creations from a particular language to connect to RL-Glue is called a codec. We currently have codecs for:

C/C++
Java
Python
Matlab
Lisp

Members of the reinforcement learning community are welcome to write their own language-or-project specific codecs to use with RL-Glue. The Lisp codec is an example of a user-contributed codec. There are currently codecs in development to connect projects as diverse as: a real-time strategy game, an atari emulator, and a robot to RL-Glue.

The RL-Glue software project, combined with the RL-Glue codecs are a powerful tool that allows members of the reinforcement learning community to re-use each others agents, environments, and experiment programs to help quicken the pace of research. Before RL-Glue most researchers implemented their own experiment protocol, making collaboration difficult.

RL-Glue has been the base for the last few reinforcement learning competitions, and that trend will continue with the 2009 Reinforcement Learning Competition.

What's new in RL-Glue 3.0

- A new homepage: http://glue.rl-community.org/

Revamped build system (autotools) for maximum cross-platform install compatibility (Linux, Unix, MacOS, Cygwin) -Installing has never been simpler: >$ ./configure >$ make >$ sudo make install
RL-Glue now installs to /usr/local
- Headers and libs in standard search paths
  - Compiling agents/environments/experiments has never been easier: >$ gcc MyAgent.c -lrlagent -o myAgent.exe
Codecs for C/C++, Java, Python, MATLAB AND LISP <--- MATLAB AND LISP!
charArray (String) observation and action types!
Documentation
- General RL-Glue Overview
- RL-Glue Technical Manual
- A manual for each codec!

History of RL-Glue

We can trace RL-Glue back as far as 1996 through a project by Rich Sutton and Juan Carlos Santamaria called RL-Interface. Since then, the project has gone through several designs and languages. Over time the objectives of the project became more ambitious - it grew from being a convenient calling convention within a single language to a complete protocol allowing all sorts of various languages to communicate with each other.

mloss.org RL Glue and Codecs

RL Glue and Codecs -- Glue 3.x and Codecs