# A note on goal-based hierarchical RL

arXiv: https://arxiv.org/abs/2609.14605 · HTML: https://mlsanity.com/paper/2609.14605/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 13 references that are in the map.
- First posted on arXiv 13 Sep 2026.

## Builds on (most cited first, 2 of 2)

- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Collapsed Amortized Variational Inference for Switching Nonlinear Dynamical Systems](https://mlsanity.com/paper/1910.09588/) (2019)

## Abstract (arXiv)

The agent-centric general value function (ACGVF) construction of citettasse2026goal lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In citetmurphy2025rl, a general agent design was proposed where the policy is based on an internal belief state zt and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) citepmurphy2001hhmm.