mlsanityData to 7 Oct 2026

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

arXiv 2010.13611 ↗First posted 26 Oct 2020cs.LGcited by 164

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning (RL) has achieved impressive performance in a variety of online settings in which an agent's ability to query the environment for transitions and rewards is effectively unlimited. However, in many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. In this work, we focus on this offline setting. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors before using these primitives for downstream task learning. Primitives extracted in this way serve two purposes: they delineate the behaviors that are supported by the data from those that are not, making them useful for avoiding distributional shift in offline RL; and they provide a degree of temporal abstraction, which reduces the effective horizon yielding better learning in theory, and improved offline RL in practice. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning as well as exploration and transfer in online RL on a variety of benchmark domains. Visualizations are available at https://sites.google.com/view/opal-iclr

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 50 references that are in the map. First posted on arXiv 26 Oct 2020.

Builds on

35 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  12. 2016The Option-Critic Architecturecited by 433
18 more
  1. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  2. 2019Learning Latent Plans from Playcited by 310
  3. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2019Challenges of Real-World Reinforcement Learningcited by 282
  6. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  7. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  8. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  9. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  10. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  11. 2018Neural probabilistic motor primitives for humanoid controlcited by 96
  12. 2017Robust Imitation of Diverse Behaviorscited by 85
  13. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  14. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  15. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  16. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  17. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 39
  18. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 41; 2023: 30; 2024: 28; 2025: 24; 2026: 20.

164 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  7. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  8. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  9. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  10. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  11. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  12. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
38 more
  1. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  2. 2025Reinforcement Learning with Action Chunkingcited by 97
  3. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  5. 2024Foundation Policies with Hilbert Representationscited by 89
  6. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  7. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  8. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  9. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  10. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  11. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  12. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  13. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  14. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  15. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  16. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  17. 2025Horizon Reduction Makes RL Scalablecited by 55
  18. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  19. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  20. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  21. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  22. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  23. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  24. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  25. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37
  26. 2021Hierarchical Skills for Efficient Explorationcited by 37
  27. 2022Skill-based Meta-Reinforcement Learningcited by 36
  28. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  29. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  30. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  31. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  32. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  33. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  34. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  35. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  36. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  37. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  38. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28

50 most cited of 164 All 164 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2021A Minimalist Approach to Offline Reinforcement Learning
  4. 2020Offline Reinforcement Learning Hands-On
  5. 2022The Challenges of Exploration for Offline Reinforcement Learning
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2021Offline Reinforcement Learning as Anti-Exploration