mlsanityData to 7 Oct 2026

Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

arXiv 2104.07749 ↗First posted 15 Apr 2021cs.RO cs.LGcited by 125

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. The videos of our experiments can be found at https://actionable-models.github.io

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 51 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

37 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning27 Oct

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
24 more
  1. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  2. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  3. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  4. 2018Stochastic Video Generation with a Learned Priorcited by 244
  5. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  6. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  7. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  8. 2019Goal-conditioned Imitation Learningcited by 159
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  10. 2018Self-Imitation Learningcited by 135
  11. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  12. 2019Exploring Model-based Planning with Policy Networkscited by 90
  13. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  14. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  15. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  16. 2019Positive-Unlabeled Reward Learningcited by 36
  17. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  18. 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
  19. 2020Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planningcited by 26
  20. 2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
  21. 2019Reinforcement Learning without Ground-Truth Statecited by 15
  22. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  23. 2017A probabilistic data-driven model for planar pushingcited by 8
  24. 2020Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulatorscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 33; 2023: 33; 2024: 21; 2025: 15; 2026: 11.

125 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  7. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  8. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  10. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  11. 2021Discovering and Achieving Goals via World Modelscited by 131
  12. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
38 more
  1. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  2. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  3. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  4. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  5. 2024Foundation Policies with Hilbert Representationscited by 89
  6. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  7. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  8. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  9. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  10. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  11. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  12. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  13. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  14. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  15. 2023SACSoN: Scalable Autonomous Control for Social Navigationcited by 60
  16. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  17. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 55
  18. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  19. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  20. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  21. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  22. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  23. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  24. 2022Skill-based Meta-Reinforcement Learningcited by 36
  25. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  26. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  27. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  28. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  29. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  30. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  31. 2022Offline Reinforcement Learning for Visual Navigationcited by 28
  32. 2025Self-Improving Embodied Foundation Modelscited by 28
  33. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  34. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  35. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  36. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  37. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  38. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  2. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  3. 2023Robotic Offline RL from Internet Videos via Value-Function Pre-Training
  4. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback
  5. 2020Offline Reinforcement Learning from Images with Latent Space Models
  6. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  7. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
  8. 2022Implicit Offline Reinforcement Learning via Supervised Learning