# Benchmarking Reinforcement Learning Algorithms on Real-World Robots

arXiv: https://arxiv.org/abs/1809.07731 · HTML: https://mlsanity.com/paper/1809.07731/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2018.

## Built on by (most cited first, 30 of 65)

- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2305.16589/) (2023)
- [SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning](https://mlsanity.com/paper/2108.13035/) (2021)
- [Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning](https://mlsanity.com/paper/2309.11359/) (2023)
- [Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation](https://mlsanity.com/paper/2307.07907/) (2023)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2110.03375/) (2021)
- [Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers](https://mlsanity.com/paper/2411.15370/) (2024)
- [Quantum Deep Reinforcement Learning for Robot Navigation Tasks](https://mlsanity.com/paper/2202.12180/) (2022)
- [Low-Rank Modular Reinforcement Learning via Muscle Synergy](https://mlsanity.com/paper/2210.15479/) (2022)
- [SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning](https://mlsanity.com/paper/2102.11075/) (2021)
- [Real-Time Reasoning Agents in Evolving Environments](https://mlsanity.com/paper/2511.04898/) (2025)
- [Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers](https://mlsanity.com/paper/2210.02317/) (2022)
- [Reinforcement Learning from Delayed Observations via World Models](https://mlsanity.com/paper/2403.12309/) (2024)
- [Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning](https://mlsanity.com/paper/2407.00324/) (2024)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning](https://mlsanity.com/paper/2303.14716/) (2023)
- [Hierarchical Reinforcement Learning for Power Network Topology Control](https://mlsanity.com/paper/2311.02129/) (2023)
- [MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2312.15339/) (2023)
- [Model-free Policy Learning with Reward Gradients](https://mlsanity.com/paper/2103.05147/) (2021)
- [An Alternate Policy Gradient Estimator for Softmax Policies](https://mlsanity.com/paper/2112.11622/) (2021)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Offline Reinforcement Learning for Optimizing Production Bidding Policies](https://mlsanity.com/paper/2310.09426/) (2023)
- [OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation](https://mlsanity.com/paper/2110.00704/) (2021)
- [Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies](https://mlsanity.com/paper/2410.16632/) (2024)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning](https://mlsanity.com/paper/2406.03276/) (2024)
- [Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy](https://mlsanity.com/paper/2511.09737/) (2025)
- [FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement](https://mlsanity.com/paper/2607.01111/) (2026)

## Abstract (arXiv)

Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.