mlsanityData to 7 Oct 2026

Provable Hierarchy-Based Meta-Reinforcement Learning

arXiv 2110.09507 ↗First posted 18 Oct 2021cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) has seen widespread interest as an approach to tractable learning of complex modular behaviors. However, existing work either assume access to expert-constructed hierarchies, or use hierarchy-learning heuristics with no provable guarantees. To address this gap, we analyze HRL in the meta-RL setting, where a learner learns latent hierarchical structure during meta-training for use in a downstream task. We consider a tabular setting where natural hierarchical structure is embedded in the transition dynamics. Analogous to supervised meta-learning theory, we provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Furthermore, we provide regret bounds on a learner using the recovered hierarchy to solve a meta-test task. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

21 of 40 references.

References · 21

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  8. 2020First return, then explorecited by 256
  9. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  10. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  11. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  12. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
9 more
  1. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  2. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  3. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 104
  4. 2017Meta Learning Shared Hierarchiescited by 93
  5. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  6. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  7. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  8. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  9. 2017Exploration--Exploitation in MDPs with Optionscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  2. 2025Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learningcited by 5
  3. 2022Efficient Meta Reinforcement Learning for Preference-based Fast Adaptationcited by 4
  4. 2024Hierarchical Subspaces of Policies for Continual Offline Reinforcement Learningcited by 2
  5. 2026Provably Safe Sim-to-Real Transfercited by 0