RLLib Lightweight On or Off Policy Reinforcement Learning Library 2.0

Saminda Abeyruwan — Fri, 25 Apr 2014 02:58:32 -0000

RLLib

(C++ Template Library to Predict, Control, Learn Behaviors, and Represent Learnable Knowledge using On/Off Policy Reinforcement Learning)

RLLib is a lightweight C++ template library that implements incremental, standard, and gradient temporal-difference learning algorithms in Reinforcement Learning. It is an optimized library for robotic applications and embedded devices that operates under fast duty cycles (e.g., < 30 ms). RLLib has been tested and evaluated on RoboCup 3D soccer simulation agents, physical NAO V4 humanoid robots, and Tiva C series launchpad microcontrollers to predict, control, learn behaviors, and represent learnable knowledge. The implementation of the RLLib library is inspired by the RLPark API, which is a library of temporal-difference learning algorithms written in Java.

Features

Off-policy prediction algorithms:
GTD(lambda)
GQ(lambda)
Off-policy control algorithms:
Greedy-GQ(lambda)
Softmax-GQ(lambda)
Off-PAC (can be used in on-policy setting)
On-policy algorithms:
TD(lambda)
TD(lambda)AlphaBound
TD(lambda)True
Sarsa(lambda)
Sarsa(lambda)AlphaBound
Sarsa(lambda)True
Sarsa(lambda)Expected
Actor-Critic (continuous actions, discrete actions, discounted reward settting, averaged reward settings, and so on)
Supervised learning algorithms:
Adaline
IDBD
KI
SemiLinearIDBD
Autostep
Policies: Random RandomX%Bias Greedy Epsilon-greedy Boltzmann Normal Softmax
Dot product: An efficient implementation of the dot product for tile coding based feature representations (with culling traces).
Benchmarking environments: Mountain Car Mountain Car 3D Swinging Pendulum Continuous Grid World Bicycle Cart Pole Acrobot Non-Markov Pole Balancing Helicopter
Optimization: Optimized for very fast duty cycles (e.g., with culling traces, RLLib has been tested on the Robocup 3D simulator agent, and on the NAO V4 (cognition thread)).
Usage: The algorithm usage is very much similar to RLPark, therefore, swift learning curve.
Examples: There are a plethora of examples demonstrating on-policy and off-policy control experiments.
Visualization: We provide a Qt4 based application to visualize benchmark problems.

Extension

Extension for Tiva C Series EK-TM4C123GXL LaunchPad, and Tiva C Series TM4C129 Connected LaunchPad microcontrollers.

Tiva C series launchpad microcontrollers: https://github.com/samindaa/csc688

Demo

Usage

RLLib is a C++ template library. The header files are located in the src directly. You can simply include this directory from your projects, e.g., -I./src, to access the algorithms.

To access the control algorithms:

#include "ControlAlgorithm.h"

To access the predication algorithms:

#include "PredictorAlgorithm"

To access the supervised learning algorithms:

#include "SupervisedAlgorithm.h"

RLLib uses the namespace:

using namespace RLLib

Testing

RLLib provides a flexible testing framework. Follow these steps to quickly write a test case.

To access the testing framework: #include "HeaderTest.h"

`javascript

include "HeaderTest.h"

RLLIB_TEST(YourTest)

class YourTest Test: public YourTestBase { public: YourTestTest() {}

virtual ~Test() {}
void run();

private: void testYourMethod(); };

void YourTestBase::testYourMethod() {/* Your test code /}

void YourTestBase::run() { testYourMethod(); } `

Add YourTest to the test/test.cfg file.
You can use @YourTest to execute only YourTest. For example, if you need to execute only MountainCar test cases, use @MountainCarTest.

Test Configuration

The test cases are executed using:

64-bit machines:
- ./configure_m64
- make
- ./RLLibTest
32-bit machines:
- ./configure_m32
- make
- ./RLLibTest
Debugging:
- ./configure_debug
- make
- ./RLLibTest

Visualization

RLLib provides a QT4.8 based Reinforcement Learning problems and algorithms visualization tool named RLLibViz. Currently RLLibViz visualizes following problems and algorithms:

On-policy:
- SwingPendulum problem with continuous actions. We use AverageRewardActorCritic algorithm.
Off-policy:
- ContinuousGridworld and MountainCar problems with discrete actions. We use Off-PAC algorithm.
In order to run the visualization tool, you need to have QT4.8 installed in your system.
In order to install RLLibViz:
- Change directory to visualization/RLLibViz
- ./configure
- ./RLLibVizSwingPendulum
- ./RLLibVizContinuousGridworld
- ./RLLibVizMountainCar

Documentation

Publication

Dynamic Role Assignment using General ValueFunctions

Contact

Saminda Abeyruwan (saminda@cs.miami.edu)

mloss.org RLLib Lightweight On or Off Policy Reinforcement Learning Library