mlsanityData to 13 Oct 2026

Hierarchical Reinforcement Learning By Discovering Intrinsic Options

arXiv 2101.06521 ↗First posted 16 Jan 2021cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks. Unlike current hierarchical RL approaches that tend to formulate goal-reaching low-level tasks or pre-define ad hoc lower-level policies, HIDIO encourages lower-level option learning that is independent of the task at hand, requiring few assumptions or little knowledge about the task structure. These options are learned through an intrinsic entropy minimization objective conditioned on the option sub-trajectories. The learned options are diverse and task-agnostic. In experiments on sparse-reward robotic manipulation and navigation tasks, HIDIO achieves higher success rates with greater sample efficiency than regular RL baselines and two state-of-the-art hierarchical RL methods.

Key facts. Cited by 57 papers in the map (data through 13 Oct 2026). Builds on 30 of its 67 references that are in the map. First posted on arXiv 16 Jan 2021.

Builds on

30 of 67 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2015Prioritized Experience Replaycited by 1,078
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 754
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 739
  6. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 442
  7. 2016The Option-Critic Architecturecited by 439
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 332
  9. 2019Learning Latent Plans from Playcited by 313
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 309
  11. 2016Variational Intrinsic Controlcited by 264
  12. 2020Revisiting Fundamentals of Experience Replaycited by 173
18 more
  1. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 165
  2. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  3. 2018Variational Option Discovery Algorithmscited by 121
  4. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  5. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  6. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  7. 2018Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependenciescited by 43
  8. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Informationcited by 41
  9. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33
  10. 2017Efficient Parallel Methods for Deep Reinforcement Learningcited by 32
  11. 2018Learning Abstract Optionscited by 31
  12. 2018Hierarchical visuomotor control of humanoidscited by 29
  13. 2017Learnings Options End-to-End for Continuous Action Taskscited by 24
  14. 2018Learning Representations in Model-Free Hierarchical Reinforcement Learningcited by 13
  15. 2018Safe Option-Critic: Learning Safety in the Option-Critic Architecturecited by 10
  16. 2019Hypothesis-Driven Skill Discovery for Hierarchical Deep Reinforcement Learningcited by 8
  17. 2018Natural Option Criticcited by 6
  18. 2019Planning with Abstract Learned Models While Learning Transferable Subtaskscited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 10; 2022: 15; 2023: 12; 2024: 7; 2025: 8; 2026: 5.

57 papers.

Sort:

By area:

  1. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 174
  2. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  3. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 82
  4. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 70
  5. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 61
  6. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  7. 2026Hierarchical Planning with Latent World Modelscited by 40
  8. 2021Hierarchical Skills for Efficient Explorationcited by 37
  9. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  10. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  11. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 27
  12. 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
38 more
  1. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  2. 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
  3. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  4. 2021HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanismcited by 19
  5. 2021Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reachingcited by 19
  6. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  7. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  8. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  9. 2021TAAC: Temporally Abstract Actor-Critic for Continuous Controlcited by 14
  10. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  11. 2022Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimizationcited by 12
  12. 2023Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Modelscited by 9
  13. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  14. 2021Program Synthesis Guided Reinforcement Learning for Partially Observed Environmentscited by 8
  15. 2023Learning to Discover Skills through Guidancecited by 8
  16. 2022Self-Activating Neural Ensembles for Continual Reinforcement Learningcited by 7
  17. 2023Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learningcited by 7
  18. 2025HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searchescited by 7
  19. 2022SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Explorationcited by 6
  20. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  21. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  22. 2022Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Taskscited by 4
  23. 2021HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learningcited by 3
  24. 2022Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planningcited by 3
  25. 2025Evaluating Interpretable Reinforcement Learning by Distilling Policies into Programscited by 3
  26. 2023Machine Learning Meets Advanced Robotic Manipulationcited by 2
  27. 2026Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policiescited by 2
  28. 2022A Survey on Deep Reinforcement Learning-based Approaches for Adaptation and Generalizationcited by 1
  29. 2022CIM: Constrained Intrinsic Motivation for Sparse-Reward Continuous Controlcited by 1
  30. 2024SOAP-RL: Sequential Option Advantage Propagation for Reinforcement Learning in POMDP Environmentscited by 1
  31. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approachcited by 1
  32. 2025The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systemscited by 1
  33. 2022Safer Autonomous Driving in a Stochastic, Partially-Observable Environment by Hierarchical Contingency Planningcited by 0
  34. 2022Emergency action termination for immediate reaction in hierarchical reinforcement learningcited by 0
  35. 2022Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agentscited by 0
  36. 2022SHIRO: Soft Hierarchical Reinforcement Learningcited by 0
  37. 2023Partitioning Distributed Compute Jobs with Reinforcement Learning and Graph Neural Networkscited by 0
  38. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0

50 most cited of 57 All 57 (JSON)