mlsanityData to 14 Oct 2026

Action-Sufficient Goal Representations

arXiv 2601.22496 ↗First posted 30 Jan 2026cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In offline goal-conditioned reinforcement learning (GCRL), hierarchical approaches decompose long-horizon tasks into high-level subgoal prediction and low-level action execution. A critical design choice in such architectures is the goal representation-the compressed encoding of goals that serves as the interface between these levels. Existing methods derive this representation from value learning, implicitly assuming that information sufficient for value estimation is adequate for optimal action prediction. We show that this assumption can fail even under exact value estimation, as such representations may collapse goals requiring distinct optimal actions. To address this, we introduce action sufficiency, an information-theoretic condition on goal representations necessary for optimal action prediction. We prove that value sufficiency, the preservation of sufficient information for value estimation, does not imply action sufficiency and empirically verify that the latter is more strongly associated with control success in a discrete environment. We further demonstrate that an actor-based representation, naturally induced by standard log-likelihood training of the low-level policy, is approximately action-sufficient. Empirically, our actor-based representations consistently outperform representations learned via value function estimation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 55 of its 66 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

55 of 66 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Dual Goal Representations8 Oct
  2. 2025Reinforcement Learning with Action Chunking10 Jul
  3. 2025Horizon Reduction Makes RL Scalable4 Jun
  4. 2025Horizon Generalization in Reinforcement Learning6 Jan
  5. 2024Learning Action-based Representations Using Invariance25 Mar
  6. 2024Foundation Policies with Hilbert Representations23 Feb

Other references · 49

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  10. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  11. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  12. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
37 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  3. 2019Goal-conditioned Imitation Learningcited by 162
  4. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
  5. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  6. 2019Planning with Goal-Conditioned Policiescited by 149
  7. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  8. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  12. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  13. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  14. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  15. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69
  16. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  17. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  18. 2019Exploration via Hindsight Goal Generationcited by 54
  19. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  20. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  21. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  22. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  23. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  24. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 31
  25. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 29
  26. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
  27. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  28. 2021Hierarchical Reinforcement Learning with Timed Subgoalscited by 21
  29. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  30. 2025Decoupled Q-Chunkingcited by 20
  31. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 16
  32. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  33. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  34. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
  35. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  36. 2022Bilinear value networkscited by 9
  37. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026WONDER: A Radio World Model-based Negotiation Framework for Multi-Agent UAV Coverage Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?
  2. 2026MSPR: Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learning
  3. 2026Dual Advantage Fields
  4. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  5. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRL
  6. 2026Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL
  7. 2026Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization
  8. 2026Unified Policy Value Decomposition for Rapid Adaptation