# Benchmarking Offline Reinforcement Learning on Real-Robot Hardware

arXiv: https://arxiv.org/abs/2307.15690 · HTML: https://mlsanity.com/paper/2307.15690/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 55 of its 82 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Placed in

- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections](https://mlsanity.com/paper/1906.04733/) (2019)

## Built on by (most cited first, 23 of 23)

- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins](https://mlsanity.com/paper/2504.13059/) (2025)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)](https://mlsanity.com/paper/2409.02920/) (2024)
- [DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.05333/) (2023)
- [Scaling Offline RL via Efficient and Expressive Shortcut Models](https://mlsanity.com/paper/2505.22866/) (2025)
- [Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization](https://mlsanity.com/paper/2311.03351/) (2023)
- [ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2309.03081/) (2023)
- [ReCoRe: Regularized Contrastive Representation Learning of World Model](https://mlsanity.com/paper/2312.09056/) (2023)
- [Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies](https://mlsanity.com/paper/2410.16632/) (2024)
- [Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy](https://mlsanity.com/paper/2605.13435/) (2026)
- [An Invitation to Deep Reinforcement Learning](https://mlsanity.com/paper/2312.08365/) (2023)
- [Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World](https://mlsanity.com/paper/2308.07741/) (2023)
- [A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2309.16718/) (2023)
- [Goal-conditioned Offline Planning from Curious Exploration](https://mlsanity.com/paper/2311.16996/) (2023)
- [Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach](https://mlsanity.com/paper/2312.07795/) (2023)
- [Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spaces](https://mlsanity.com/paper/2402.10820/) (2024)
- [Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning](https://mlsanity.com/paper/2505.20621/) (2025)
- [Dataset Clustering for Improved Offline Policy Learning](https://mlsanity.com/paper/2402.09550/) (2024)
- [A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Robotics](https://mlsanity.com/paper/2411.05718/) (2024)
- [Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations](https://mlsanity.com/paper/2412.18781/) (2024)
- [Greener Deep Reinforcement Learning: Analysis of Energy and Carbon Efficiency Across Atari Benchmarks](https://mlsanity.com/paper/2509.05273/) (2025)
- [Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control](https://mlsanity.com/paper/2510.13358/) (2025)
- [StagePilot: Stage-Level Planning for Long-Horizon Dialogue Simulation in Cybergrooming](https://mlsanity.com/paper/2602.05060/) (2026)

## Abstract (arXiv)

Learning policies from previously recorded data is a promising direction for real-world robotics tasks, as online learning is often infeasible. Dexterous manipulation in particular remains an open problem in its general form. The combination of offline reinforcement learning with large diverse datasets, however, has the potential to lead to a breakthrough in this challenging domain analogously to the rapid progress made in supervised learning in recent years. To coordinate the efforts of the research community toward tackling this problem, we propose a benchmark including: i) a large collection of data for offline learning from a dexterous manipulation platform on two tasks, obtained with capable RL agents trained in simulation; ii) the option to execute learned policies on a real-world robotic system and a simulation for efficient debugging. We evaluate prominent open-sourced offline reinforcement learning algorithms on the datasets and provide a reproducible experimental setup for offline reinforcement learning on real systems.