mlsanityData to 14 Oct 2026

Real World Offline Reinforcement Learning with Realistic Data Source

arXiv 2210.06479 ↗First posted 12 Oct 2022cs.RO cs.LGcited by 17

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Offline reinforcement learning (ORL) holds great promise for robot learning due to its ability to learn from arbitrary pre-generated experience. However, current ORL benchmarks are almost entirely in simulation and utilize contrived datasets like replay buffers of online RL agents or sub-optimal trajectories, and thus hold limited relevance for real-world robotics. In this work (Real-ORL), we posit that data collected from safe operations of closely related tasks are more practical data sources for real-world robot learning. Under these settings, we perform an extensive (6500+ trajectories collected over 800+ robot hours and 270+ human labor hour) empirical study evaluating generalization and transfer capabilities of representative ORL methods on four real-world tabletop manipulation tasks. Our study finds that ORL and imitation learning prefer different action spaces, and that ORL algorithms can generalize from leveraging offline heterogeneous data sources and outperform imitation learning. We release our dataset and implementations at URL: https://sites.google.com/view/real-orl

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 59 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

51 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Offline Meta-Reinforcement Learning for Industrial Insertion8 Oct
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulation6 Aug
  3. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills15 Apr
  4. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning27 Oct

Other references · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
35 more
  1. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  7. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  8. 2021Implicit Behavioral Cloningcited by 452
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  10. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  13. 2020Critic Regularized Regressioncited by 275
  14. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  15. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  16. 2019A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithmscited by 142
  17. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  18. 2020Model-Based Offline Planningcited by 129
  19. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  20. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  21. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  22. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 73
  23. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  24. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  25. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  26. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  27. 2021STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulationcited by 47
  28. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  29. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  30. 2020Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulationcited by 38
  31. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  32. 2021Active Offline Policy Selectioncited by 26
  33. 2020Semi-supervised reward learning for offline reinforcement learningcited by 23
  34. 2022RB2: Robotic Manipulation Benchmarking with a Twistcited by 23
  35. 2021Offline Inverse Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 4; 2025: 3; 2026: 4.

17 papers.

Sort:

By area:

  1. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 29
  2. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  3. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  4. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 23
  5. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  6. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  7. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  8. 2024Scaling Vision-and-Language Navigation With Offline RLcited by 5
  9. 2023Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environmentscited by 4
  10. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  11. 2024Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learningcited by 2
  12. 2026Inference Time Policy Optimization for Offline RL with Differentiable World Modelscited by 1
5 more
  1. 2026Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learningcited by 1
  2. 2024Disentangled Representations for Causal Cognitioncited by 0
  3. 2025RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulationcited by 0
  4. 2026120 Minutes and a Laptop: Minimalist Image-goal Navigation via Unsupervised Exploration and Offline RLcited by 0
  5. 2026Robustness of Robotic Manipulation: Foundations and Frontierscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  2. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
  3. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  4. 2023Finetuning Offline World Models in the Real World
  5. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning
  6. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies