# Hierarchical Policy Learning is Sensitive to Goal Space Design

arXiv: https://arxiv.org/abs/1905.01537 · HTML: https://mlsanity.com/paper/1905.01537/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2019.

## Built on by (most cited first, 8 of 8)

- [Active Hierarchical Exploration with Stable Subgoal Representation Learning](https://mlsanity.com/paper/2105.14750/) (2021)
- [Self-supervised Reinforcement Learning with Independently Controllable Subgoals](https://mlsanity.com/paper/2109.04150/) (2021)
- [Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning](https://mlsanity.com/paper/2211.00247/) (2022)
- [E-MAPP: Efficient Multi-Agent Reinforcement Learning with Parallel Program Guidance](https://mlsanity.com/paper/2212.02064/) (2022)
- [Adaptive trajectory-constrained exploration strategy for deep reinforcement learning](https://mlsanity.com/paper/2312.16456/) (2023)
- [Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery](https://mlsanity.com/paper/2412.11930/) (2024)
- [CORD: Generalizable Cooperation via Role Diversity](https://mlsanity.com/paper/2501.02221/) (2025)
- [Solving Sokoban using Hierarchical Reinforcement Learning with Landmarks](https://mlsanity.com/paper/2504.04366/) (2025)

## Abstract (arXiv)

Hierarchy in reinforcement learning agents allows for control at multiple time scales yielding improved sample efficiency, the ability to deal with long time horizons and transferability of sub-policies to tasks outside the training distribution. It is often implemented as a master policy providing goals to a sub-policy. Ideally, we would like the goal-spaces to be learned, however, properties of optimal goal spaces still remain unknown and consequently there is no method yet to learn optimal goal spaces. Motivated by this, we systematically analyze how various modifications to the ground-truth goal-space affect learning in hierarchical models with the aim of identifying important properties of optimal goal spaces. Our results show that, while rotation of ground-truth goal spaces and noise had no effect, having additional unnecessary factors significantly impaired learning in hierarchical models.