Hierarchical Reinforcement Learning in Complex 3D Environments
Not on the map. Why?
Abstract (arXiv)
Hierarchical Reinforcement Learning (HRL) agents have the potential to demonstrate appealing capabilities such as planning and exploration with abstraction, transfer, and skill reuse. Recent successes with HRL across different domains provide evidence that practical, effective HRL agents are possible, even if existing agents do not yet fully realize the potential of HRL. Despite these successes, visually complex partially observable 3D environments remained a challenge for HRL agents. We address this issue with Hierarchical Hybrid Offline-Online (H2O2), a hierarchical deep reinforcement learning agent that discovers and learns to use options from scratch using its own experience. We show that H2O2 is competitive with a strong non-hierarchical Muesli baseline in the DeepMind Hard Eight tasks and we shed new light on the problem of learning hierarchical agents in complex environments. Our empirical study of H2O2 reveals previously unnoticed practical challenges and brings new perspective to the current understanding of hierarchical agents in complex domains.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 28 Feb 2023.
Builds on
33 of 53 references.
References · 33
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
- 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
- 2017Hindsight Experience Replaycited by 1,096
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
- 2019Learning Latent Plans from Playcited by 310
21 more
- 2016Variational Intrinsic Controlcited by 262
- 2018Distributed Prioritized Experience Replaycited by 260
- 2019Efficient Exploration via State Marginal Matchingcited by 186
- 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
- 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
- 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
- 2021Discovering and Achieving Goals via World Modelscited by 131
- 2018Neural probabilistic motor primitives for humanoid controlcited by 96
- 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
- 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
- 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
- 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
- 2021Regularized Behavior Value Estimationcited by 32
- 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
- 2021Reverb: A Framework For Experience Replaycited by 27
- 2021Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoningcited by 25
- 2019The Termination Criticcited by 24
- 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22
- 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
- 2020Using Unity to Help Solve Intelligencecited by 13
- 2016Learning Purposeful Behaviour in the Absence of Rewardscited by 6
Built on by
Not cited yet.