mlsanityData to 7 Oct 2026

Near-Optimal Representation Learning for Hierarchical Reinforcement Learning

arXiv 1810.01257 ↗First posted 2 Oct 2018cs.AIcited by 107

Not on the map. Why?

Abstract (arXiv)

We study the problem of representation learning in goal-conditioned hierarchical reinforcement learning. In such hierarchical structures, a higher-level controller solves tasks by iteratively communicating goals which a lower-level policy is trained to reach. Accordingly, the choice of representation -- the mapping of observation space to goal space -- is crucial. To study this problem, we develop a notion of sub-optimality of a representation, defined in terms of expected reward of the optimal hierarchical policy using this representation. We derive expressions which bound the sub-optimality and show how these expressions can be translated to representation learning objectives which may be optimized in practice. Results on a number of difficult continuous-control tasks show that our approach to representation learning yields qualitatively better representations as well as quantitatively better hierarchical policies, compared to existing methods (see videos at https://sites.google.com/view/representation-hrl).

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 31; 2022: 21; 2023: 19; 2024: 16; 2025: 12; 2026: 6.

107 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  4. 2022Deep Hierarchical Planning from Pixelscited by 96
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  6. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  7. 2019State Alignment-based Imitation Learningcited by 73
  8. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  9. 2025Horizon Reduction Makes RL Scalablecited by 55
  10. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  11. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 46
  12. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 42
38 more
  1. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  2. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  3. 2021Hierarchical Skills for Efficient Explorationcited by 37
  4. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  5. 2021Robust Predictable Controlcited by 29
  6. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  7. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  8. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  9. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  10. 2022Causality-driven Hierarchical Structure Discovery for Reinforcement Learningcited by 23
  11. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  12. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 20
  13. 2022Language Control Diffusion: Efficiently Scaling through Space, Time, and Taskscited by 20
  14. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  15. 2021Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Informationcited by 18
  16. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 18
  17. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  18. 2021Active Hierarchical Exploration with Stable Subgoal Representation Learningcited by 17
  19. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  20. 2021Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learningcited by 15
  21. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  22. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  23. 2021Motion Planning by Learning the Solution Manifold in Trajectory Optimizationcited by 13
  24. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  25. 2021Learning Domain Invariant Representations in Goal-conditioned Block MDPscited by 12
  26. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  27. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  28. 2021Hierarchies of Planning and Reinforcement Learning for Robot Navigationcited by 11
  29. 2023NetHack is Hard to Hackcited by 11
  30. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10
  31. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  32. 2024Towards Principled Representation Learning from Videos for Reinforcement Learningcited by 10
  33. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learningcited by 9
  34. 2024Learning Multi-Agent Loco-Manipulation for Long-Horizon Quadrupedal Pushingcited by 9
  35. 2021Multi-Task Learning with Sequence-Conditioned Transporter Networkscited by 8
  36. 2021Context-Specific Representation Abstraction for Deep Option Learningcited by 8
  37. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  38. 2022ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learningcited by 8

50 most cited of 107 All 107 (JSON)