mlsanityData to 7 Oct 2026

A Workflow for Offline Model-Free Robotic Reinforcement Learning

arXiv 2109.10813 ↗First posted 22 Sep 2021cs.LGcited by 68

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) enables learning control policies by utilizing only prior experience, without any online interaction. This can allow robots to acquire generalizable skills from large and diverse datasets, without any costly or unsafe online data collection. Despite recent algorithmic advances in offline RL, applying these methods to real-world problems has proven challenging. Although offline RL methods can learn from prior data, there is no clear and well-understood process for making various design choices, from model architecture to algorithm hyperparameters, without actually evaluating the learned policies online. In this paper, our aim is to develop a practical workflow for using offline RL analogous to the relatively well-understood workflows for supervised learning problems. To this end, we devise a set of metrics and conditions that can be tracked over the course of offline training, and can inform the practitioner about how the algorithm and model architecture should be adjusted to improve final performance. Our workflow is derived from a conceptual understanding of the behavior of conservative offline RL algorithms and cross-validation in supervised learning. We demonstrate the efficacy of this workflow in producing effective policies without any online tuning, both in several simulated robotic learning scenarios and for three tasks on two distinct real robots, focusing on learning manipulation skills with raw image observations with sparse binary rewards. Explanatory video and additional results can be found at sites.google.com/view/offline-rl-workflow

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 67 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

55 of 67 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  5. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2016Deep Variational Information Bottleneckcited by 915
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  12. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
32 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  8. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  9. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  10. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  11. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  12. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  13. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  14. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  15. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  16. 2020Visual Imitation Made Easycited by 111
  17. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  18. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  19. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  20. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  21. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  22. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  23. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresightcited by 34
  24. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  25. 2020Representations for Stable Off-Policy Reinforcement Learningcited by 31
  26. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  27. 2016Visual Closed-Loop Control for Pouring Liquidscited by 26
  28. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  29. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  30. 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22
  31. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  32. 2018Interpretable Latent Spaces for Learning from Demonstrationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 24; 2023: 18; 2024: 13; 2025: 9; 2026: 2.

68 papers.

Sort:

By area:

  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  3. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  8. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  9. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  10. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  11. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  12. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
38 more
  1. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  2. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
  3. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  4. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  5. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  6. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  7. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Trainingcited by 31
  8. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  9. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  10. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  11. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  12. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  13. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
  14. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  15. 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Datasetcited by 14
  16. 2022Bayesian Imitation Learning for End-to-End Mobile Manipulationcited by 13
  17. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  18. 2022Learning Robotic Navigation from Experience: Principles, Methods, and Recent Resultscited by 13
  19. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  20. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 12
  21. 2021Model Selection in Batch Policy Optimizationcited by 11
  22. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11
  23. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  24. 2023A Primal-Dual-Critic Algorithm for Offline Constrained Reinforcement Learningcited by 9
  25. 2022Learning Value Functions from Undirected State-only Experiencecited by 8
  26. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learningcited by 8
  27. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  28. 2025OmniVLA: Physically-Grounded Multimodal VLA with Unified Multi-Sensor Perception for Robotic Manipulationcited by 7
  29. 2024Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replaycited by 6
  30. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  31. 2024Offline Reinforcement Learning for Learning to Dispatch for Job Shop Schedulingcited by 6
  32. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  33. 2023Model-based Offline Reinforcement Learning with Local Misspecificationcited by 5
  34. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  35. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  36. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  37. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  38. 2022Offline Reinforcement Learning at Multiple Frequenciescited by 4

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Model-Based Offline Planning
  2. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  3. 2022Implicit Offline Reinforcement Learning via Supervised Learning
  4. 2023Finetuning Offline World Models in the Real World
  5. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
  6. 2023Deploying Offline Reinforcement Learning with Human Feedback
  7. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory