mlsanityData to 7 Oct 2026

Discovery of Options via Meta-Learned Subgoals

arXiv 2102.06741 ↗First posted 12 Feb 2021cs.LG cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager-worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both a set of task-independent discovered-options and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extended options in multi-task RL domains, (2) the discovered options are frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help a randomly initialised manager learn faster in completely new tasks.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

31 of 55 references.

References · 31

  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  6. 2016The Option-Critic Architecturecited by 433
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  8. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  9. 2016Variational Intrinsic Controlcited by 262
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  11. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
19 more
  1. 2018Meta-Gradient Reinforcement Learningcited by 156
  2. 2017Deep Exploration via Randomized Value Functionscited by 155
  3. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  4. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  5. 2017Meta Learning Shared Hierarchiescited by 93
  6. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  7. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  8. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  9. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  10. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  11. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  12. 2020Relative Variational Intrinsic Controlcited by 34
  13. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  14. 2019The Termination Criticcited by 24
  15. 2016Adaptive Skills, Adaptive Partitions (ASAP)cited by 11
  16. 2019How Should an Agent Practice?cited by 8
  17. 2016Learning Purposeful Behaviour in the Absence of Rewardscited by 6
  18. 2020A Self-Tuning Actor-Critic Algorithmcited by 6
  19. 2019Asynchronous Coagent Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 8; 2023: 4; 2024: 4; 2025: 5; 2026: 2.

27 papers.

Sort:

By area:

  1. 2022Deep Hierarchical Planning from Pixelscited by 96
  2. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  3. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  4. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  5. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  6. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 11
  7. 2022Meta-Gradients in Non-Stationary Environmentscited by 11
  8. 2022Continual Auxiliary Task Learningcited by 9
  9. 2021Average-Reward Learning and Planning with Optionscited by 8
  10. 2024A Survey Analyzing Generalization in Deep Reinforcement Learningcited by 8
  11. 2022An Investigation of the Bias-Variance Tradeoff in Meta-Gradientscited by 4
  12. 2021Hierarchical Representation Learning for Markov Decision Processescited by 3
15 more
  1. 2022Toward Discovering Options that Achieve Faster Planningcited by 3
  2. 2024Meta SAC-Lag: Towards Deployable Safe Reinforcement Learning via MetaGradient-based Hyperparameter Tuningcited by 3
  3. 2023Scaling Goal-based Exploration via Pruning Proto-goalscited by 2
  4. 2025OptionZero: Planning with Learned Optionscited by 2
  5. 2025MOSAIC: Skill-Centric Manipulation Planning with Physics Simulationcited by 2
  6. 2023Iterative Option Discovery for Planning, by Planningcited by 1
  7. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  8. 2025Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Reviewcited by 1
  9. 2025Multi-layer Abstraction for Nested Generation of Options (MANGO) in Hierarchical Reinforcement Learningcited by 1
  10. 2026Multi-Agent Transactive Memorycited by 1
  11. 2022Reusable Options through Gradient-based Meta Learningcited by 0
  12. 2023Int-HRL: Towards Intention-based Hierarchical Reinforcement Learningcited by 0
  13. 2024On the benefits of pixel-based hierarchical policies for task generalizationcited by 0
  14. 2024Synthesizing Evolving Symbolic Representations for Autonomous Systemscited by 0
  15. 2026Mesh-RL: Coupled subgrid reinforcement learningcited by 0