mlsanityData to 7 Oct 2026

Flexible Option Learning

arXiv 2112.03097 ↗First posted 6 Dec 2021cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Temporal abstraction in reinforcement learning (RL), offers the promise of improving generalization and knowledge transfer in complex environments, by propagating information more efficiently over time. Although option learning was initially formulated in a way that allows updating many options simultaneously, using off-policy, intra-option learning (Sutton, Precup & Singh, 1999), many of the recent hierarchical reinforcement learning approaches only update a single option at a time: the option currently executing. We revisit and extend intra-option learning in the context of deep reinforcement learning, in order to enable updating all options consistent with current primitive action choices, without introducing any additional estimates. Our method can therefore be naturally adopted in most hierarchical RL frameworks. When we combine our approach with the option-critic algorithm for option discovery, we obtain significant improvements in performance and data-efficiency across a wide variety of domains.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 31 of its 69 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

31 of 69 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  6. 2016The Option-Critic Architecturecited by 433
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  8. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  9. 2019Self-Supervised Exploration via Disagreementcited by 263
  10. 2016Variational Intrinsic Controlcited by 262
  11. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  12. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
19 more
  1. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  2. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  3. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  4. 2016Q(λ) with Off-Policy Correctionscited by 48
  5. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  6. 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 36
  7. 2018Learning Abstract Optionscited by 31
  8. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  9. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  10. 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
  11. 2017Learnings Options End-to-End for Continuous Action Taskscited by 24
  12. 2015Emphatic Temporal-Difference Learningcited by 16
  13. 2020Reward Propagation Using Graph Convolutional Networkscited by 15
  14. 2017Expected Policy Gradientscited by 14
  15. 2021Temporally Abstract Partial Modelscited by 11
  16. 2018Safe Option-Critic: Learning Safety in the Option-Critic Architecturecited by 10
  17. 2017Learning with Options that Terminate Off-Policycited by 6
  18. 2018Natural Option Criticcited by 6
  19. 2019All-Action Policy Gradient Methods: A Numerical Integration Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 3; 2025: 5; 2026: 5.

24 papers.

Sort:

By area:

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  2. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
  3. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  4. 2023Generalized Animal Imitator: Agile Locomotion with Versatile Motion Priorcited by 17
  5. 2022MO2: Model-Based Offline Optionscited by 10
  6. 2023LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Frameworkcited by 7
  7. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  8. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrationscited by 5
  9. 2025Scalable Option Learning in High-Throughput Environmentscited by 5
  10. 2023Value-aware Importance Weighting for Off-policy Reinforcement Learningcited by 4
  11. 2024Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planningcited by 4
  12. 2022Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraintscited by 2
12 more
  1. 2026Temporally Extended Mixture-of-Experts Modelscited by 2
  2. 2022Importance Sampling Placement in Off-Policy Temporal-Difference Methodscited by 1
  3. 2023Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machinescited by 1
  4. 2022Reusable Options through Gradient-based Meta Learningcited by 0
  5. 2024Temporal Abstraction in Reinforcement Learning with Offline Datacited by 0
  6. 2025Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPscited by 0
  7. 2025Learning Multi-Timescale Interventions under Safety and Resource Constraintscited by 0
  8. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0
  9. 2026Enabling Option Learning in Sparse Rewards with Hindsight Experience Replaycited by 0
  10. 2026Learning When to Cooperate Under Heterogeneous Goalscited by 0
  11. 2026Hierarchical Behaviour Spacescited by 0
  12. 2026Goal-Conditioned Agents that Learn Everything All at Oncecited by 0