mlsanityData to 7 Oct 2026

Benchmarking Offline Reinforcement Learning on Real-Robot Hardware

arXiv 2307.15690 ↗First posted 28 Jul 2023cs.LG cs.ROcited by 23

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Learning policies from previously recorded data is a promising direction for real-world robotics tasks, as online learning is often infeasible. Dexterous manipulation in particular remains an open problem in its general form. The combination of offline reinforcement learning with large diverse datasets, however, has the potential to lead to a breakthrough in this challenging domain analogously to the rapid progress made in supervised learning in recent years. To coordinate the efforts of the research community toward tackling this problem, we propose a benchmark including: i) a large collection of data for offline learning from a dexterous manipulation platform on two tasks, obtained with capable RL agents trained in simulation; ii) the option to execute learned policies on a real-world robotic system and a simulation for efficient debugging. We evaluate prominent open-sourced offline reinforcement learning algorithms on the datasets and provide a reproducible experimental setup for offline reinforcement learning on real systems.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 82 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

55 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulation6 Aug

Other references · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
38 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  12. 2018Behavioral Cloning from Observationcited by 482
  13. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  14. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  15. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  16. 2020Critic Regularized Regressioncited by 269
  17. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  18. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  19. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  20. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  21. 2021Offline RL Without Off-Policy Evaluationcited by 150
  22. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  23. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  24. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  25. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  26. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  27. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  28. 2017Time Limits in Reinforcement Learningcited by 72
  29. 2019ROBEL: Robotics Benchmarks for Learning with Low-Cost Robotscited by 70
  30. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  31. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 47
  32. 2019PyRobot: An Open-source Robotics Framework for Research and Benchmarkingcited by 39
  33. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 34
  34. 2020TriFinger: An Open-Source Robot for Learning Dexteritycited by 31
  35. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  36. 2016The Robotarium: A remotely accessible swarm robotics research testbedcited by 24
  37. 2019An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Researchcited by 18
  38. 2019REPLAB: A Reproducible Low-Cost Arm Benchmark Platform for Robotic Learningcited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 7; 2025: 5; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  2. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  3. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  4. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 27
  5. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  6. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  7. 2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learningcited by 11
  8. 2023ReCoRe: Regularized Contrastive Representation Learning of World Modelcited by 11
  9. 2023An Invitation to Deep Reinforcement Learningcited by 5
  10. 2024Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policiescited by 5
  11. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
  12. 2023Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real Worldcited by 3
11 more
  1. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  2. 2023Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approachcited by 3
  3. 2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
  4. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learningcited by 3
  5. 2023A Real-World Quadrupedal Locomotion Benchmark for Offline Reinforcement Learningcited by 2
  6. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  7. 2024A Retrospective on the Robot Air Hockey Challenge: Benchmarking Robust, Reliable, and Safe Learning Techniques for Real-world Roboticscited by 1
  8. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  9. 2025Greener Deep Reinforcement Learning: Analysis of Energy and Carbon Efficiency Across Atari Benchmarkscited by 1
  10. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  11. 2026StagePilot: Stage-Level Planning for Long-Horizon Dialogue Simulation in Cybergroomingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Real World Offline Reinforcement Learning with Realistic Data Source
  2. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  3. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
  4. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies
  5. 2023Offline Skill Generalization via Task and Motion Planning
  6. 2022Implicit Offline Reinforcement Learning via Supervised Learning
  7. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  8. 2025RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation