A note on goal-based hierarchical RL

The agent-centric general value function (ACGVF) construction of citettasse2026goal lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In citetmurphy2025rl, a general agent design was proposed where the policy is based on an internal belief state zt and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) citepmurphy2001hhmm.