Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Meta-Reinforcement Learning (Meta-RL) enables fast adaptation to new testing tasks. Despite recent advancements, it is still challenging to learn performant policies across multiple complex and high-dimensional tasks. To address this, we propose a novel architecture with three hierarchical levels for 1) learning task representations, 2) discovering task-agnostic macro-actions in an automated manner, and 3) learning primitive actions. The macro-action can guide the low-level primitive policy learning to more efficiently transition to goal states. This can address the issue that the policy may forget previously learned behavior while learning new, conflicting tasks. Moreover, the task-agnostic nature of the macro-actions is enabled by removing task-specific components from the state space. Hence, this makes them amenable to re-composition across different tasks and leads to promising fast adaptation to new tasks. Also, the prospective instability from the tri-level hierarchies is effectively mitigated by our innovative, independently tailored training schemes. Experiments in the MetaWorld framework demonstrate the improved sample efficiency and success rate of our approach compared to previous state-of-the-art methods.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 25 references that are in the map. First posted on arXiv 16 Dec 2024.
Builds on
17 of 25 references.
References · 17
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
- 2017Constrained Policy Optimizationcited by 983
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
- 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
- 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
- 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
- 2019Hindsight Credit Assignmentcited by 79
- 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
- 2022Task Adaptive Parameter Sharing for Multi-Task Learningcited by 42
5 more
- 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
- 2021Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixturecited by 12
- 2019Hierarchical Policy Learning is Sensitive to Goal Space Designcited by 8
- 2023Constrained Meta-Reinforcement Learning for Adaptable Safety Guarantee with Differentiable Convex Programmingcited by 8
- 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flowscited by 2
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Boosting Hierarchical Reinforcement Learning with Meta-Learning for Complex Task Adaptation
- 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learners
- 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
- 2025Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning
- 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimation
- 2024Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator
- 2025TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environments
- 2026Meta-RL with Shared Representations Enables Fast Adaptation in Energy Systems