mlsanityData to 14 Oct 2026

Subgoal Discovery Using a Free Energy Paradigm and State Aggregations

arXiv 2412.16687 ↗First posted 21 Dec 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) plays a major role in solving complex sequential decision-making tasks. Hierarchical and goal-conditioned RL are promising methods for dealing with two major problems in RL, namely sample inefficiency and difficulties in reward shaping. These methods tackle the mentioned problems by decomposing a task into simpler subtasks and temporally abstracting a task in the action space. One of the key components for task decomposition of these methods is subgoal discovery. We can use the subgoal states to define hierarchies of actions and also use them in decomposing complex tasks. Under the assumption that subgoal states are more unpredictable, we propose a free energy paradigm to discover them. This is achieved by using free energy to select between two spaces, the main space and an aggregation space. The model ; changes from neighboring states to a given state shows the unpredictability of a given state, and therefore it is used in this paper for subgoal discovery. Our empirical results on navigation tasks like grid-world environments show that our proposed method can be applied for subgoal discovery without prior knowledge of the task. Our proposed method is also robust to the stochasticity of environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 55 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

21 of 55 references.

References · 21

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2016The Option-Critic Architecturecited by 448
  6. 2017A Tutorial on Thompson Samplingcited by 433
  7. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  8. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  9. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  10. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  11. 2017Multi-Level Discovery of Deep Optionscited by 51
  12. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
9 more
  1. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 31
  2. 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
  3. 2019Successor Options: An Option Discovery Framework for Reinforcement Learningcited by 23
  4. 2015Information-Theoretic Bounded Rationalitycited by 21
  5. 2018Finding Options that Minimize Planning Timecited by 17
  6. 2020Diversity-Enriched Option-Criticcited by 7
  7. 2023Neural Episodic Control with State Abstractioncited by 7
  8. 2019Option Encoder: A Framework for Discovering a Policy Basis in Reinforcement Learningcited by 3
  9. 2022Toward Discovering Options that Achieve Faster Planningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Goal Space Abstraction in Hierarchical Reinforcement Learning via Reachability Analysis
  2. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  3. 2019Learning from Trajectories via Subgoal Discovery
  4. 2024Identifying Selections for Unsupervised Subtask Discovery
  5. 2025Hierarchical Reinforcement Learning with Targeted Causal Interventions
  6. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
  7. 2020Exploration by Maximizing R'enyi Entropy for Reward-Free RL Framework
  8. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learning