# RL-X: A Deep Reinforcement Learning Library (not only) for RoboCup

arXiv: https://arxiv.org/abs/2310.13396 · HTML: https://mlsanity.com/paper/2310.13396/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Randomized Ensembled Double Q-Learning: Learning Fast Without a Model](https://mlsanity.com/paper/2101.05982/) (2021)
- [Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics](https://mlsanity.com/paper/2005.04269/) (2020)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Dopamine: A Research Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.06110/) (2018)
- [Dropout Q-Functions for Doubly Efficient Reinforcement Learning](https://mlsanity.com/paper/2110.02034/) (2021)
- [EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine](https://mlsanity.com/paper/2206.10558/) (2022)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [You May Not Need Ratio Clipping in PPO](https://mlsanity.com/paper/2202.00079/) (2022)
- [Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance](https://mlsanity.com/paper/2111.09159/) (2021)

## Built on by (most cited first, 8 of 8)

- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion](https://mlsanity.com/paper/2503.08375/) (2025)
- [Multi-Embodiment Locomotion at Scale with extreme Embodiment Randomization](https://mlsanity.com/paper/2509.02815/) (2025)
- [Shape Your Body: Value Gradients for Multi-Embodiment Robot Design](https://mlsanity.com/paper/2606.00702/) (2026)
- [Massively Scaling Explicit Policy-conditioned Value Functions](https://mlsanity.com/paper/2502.11949/) (2025)
- [Bridge the Gap: Enhancing Quadruped Locomotion with Vertical Ground Perturbations](https://mlsanity.com/paper/2510.13488/) (2025)
- [Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates](https://mlsanity.com/paper/2605.11020/) (2026)
- [ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2609.36238/) (2026)

## Abstract (arXiv)

This paper presents the new Deep Reinforcement Learning (DRL) library RL-X and its application to the RoboCup Soccer Simulation 3D League and classic DRL benchmarks. RL-X provides a flexible and easy-to-extend codebase with self-contained single directory algorithms. Through the fast JAX-based implementations, RL-X can reach up to 4.5x speedups compared to well-known frameworks like Stable-Baselines3.