mlsanityData to 14 Oct 2026

MO2: Model-Based Offline Options

arXiv 2209.01947 ↗First posted 5 Sep 2022cs.LG cs.AI cs.ROcited by 10

Not on the map. Why?

Abstract (arXiv)

The ability to discover useful behaviours from past experience and transfer them to new tasks is considered a core component of natural embodied intelligence. Inspired by neuroscience, discovering behaviours that switch at bottleneck states have been long sought after for inducing plans of minimum description length across tasks. Prior approaches have either only supported online, on-policy, bottleneck state discovery, limiting sample-efficiency, or discrete state-action domains, restricting applicability. To address this, we introduce Model-Based Offline Options (MO2), an offline hindsight framework supporting sample-efficient bottleneck option discovery over continuous state-action spaces. Once bottleneck options are learnt offline over source domains, they are transferred online to improve exploration and value estimation on the transfer domain. Our experiments show that on complex long-horizon continuous control tasks with sparse, delayed rewards, MO2's properties are essential and lead to performance exceeding recent option learning methods. Additional ablations further demonstrate the impact on option predictability and credit assignment.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 5 Sep 2022.

Builds on

37 of 55 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  11. 2016The Option-Critic Architecturecited by 448
  12. 2018Maximum a Posteriori Policy Optimisationcited by 328
25 more
  1. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  2. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  3. 2016Variational Intrinsic Controlcited by 267
  4. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  6. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  7. 2018Variational Option Discovery Algorithmscited by 122
  8. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  9. 2018Neural probabilistic motor primitives for humanoid controlcited by 97
  10. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  11. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  12. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  13. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 62
  14. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  15. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  16. 2020Continuous-Discrete Reinforcement Learning for Hybrid Control in Roboticscited by 44
  17. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  18. 2018Learning Abstract Optionscited by 31
  19. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  20. 2021Flexible Option Learningcited by 25
  21. 2019The Termination Criticcited by 24
  22. 2019Successor Options: An Option Discovery Framework for Reinforcement Learningcited by 23
  23. 2018The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigationcited by 7
  24. 2020Diversity-Enriched Option-Criticcited by 7
  25. 2022Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 6; 2024: 1; 2025: 1.

10 papers.

  1. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  2. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  3. 2023Chain-of-Thought Predictive Controlcited by 18
  4. 2023Efficient Planning with Latent Diffusioncited by 12
  5. 2023Predictable MDP Abstraction for Unsupervised Model-Based RLcited by 8
  6. 2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
  7. 2023Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domainscited by 6
  8. 2025NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrationscited by 1
  9. 2022Efficient Exploration using Model-Based Quality-Diversity with Gradientscited by 0
  10. 2024Temporal Abstraction in Reinforcement Learning with Offline Datacited by 0