mlsanityData to 7 Oct 2026

HIQL: Offline Goal-Conditioned RL with Latent States as Actions

arXiv 2307.11949 ↗First posted 22 Jul 2023cs.LG cs.AI cs.ROcited by 133

Not on the map. Why?

Cited by 49 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Unsupervised pre-training has recently become the bedrock for computer vision and natural language processing. In reinforcement learning (RL), goal-conditioned RL can potentially provide an analogous self-supervised approach for making use of large quantities of unlabeled (reward-free) data. However, building effective algorithms for goal-conditioned RL that can learn directly from diverse offline data is challenging, because it is hard to accurately estimate the exact value function for faraway goals. Nonetheless, goal-reaching problems exhibit structure, such that reaching distant goals entails first passing through closer subgoals. This structure can be very useful, as assessing the quality of actions for nearby goals is typically easier than for more distant goals. Based on this idea, we propose a hierarchical algorithm for goal-conditioned RL from offline data. Using one action-free value function, we learn two policies that allow us to exploit this structure: a high-level policy that treats states as actions and predicts (a latent representation of) a subgoal and a low-level policy that predicts the action for reaching this subgoal. Through analysis and didactic examples, we show how this hierarchical decomposition makes our method robust to noise in the estimated value function. We then apply our method to offline goal-reaching benchmarks, showing that our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data. Our code is available at https://seohong.me/projects/hiql/

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 87 of its 109 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

87 of 109 references.

References · 87

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2016Layer Normalizationcited by 5,579
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  14. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  15. 2018Behavioral Cloning from Observationcited by 482
  16. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  17. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  18. 2016The Option-Critic Architecturecited by 433
  19. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  20. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  21. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  22. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  23. 2019Learning Latent Plans from Playcited by 310
  24. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  25. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  26. 2020Critic Regularized Regressioncited by 269
  27. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  28. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  29. 2018Semi-parametric Topological Memory for Navigationcited by 203
  30. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  31. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  32. 2019Goal-conditioned Imitation Learningcited by 159
  33. 2021Offline RL Without Off-Policy Evaluationcited by 150
  34. 2019Planning with Goal-Conditioned Policiescited by 144
  35. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  36. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  37. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  38. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 27; 2025: 50; 2026: 53.

133 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2024Foundation Policies with Hilbert Representationscited by 89
  3. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  4. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  5. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  6. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  7. 2025Horizon Reduction Makes RL Scalablecited by 55
  8. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  9. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  10. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
38 more
  1. 2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 37
  2. 2026Hierarchical Planning with Latent World Modelscited by 34
  3. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  4. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  5. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  6. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  7. 2025Value-guided action planning with JEPA world modelscited by 19
  8. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  9. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  10. 2025Horizon Generalization in Reinforcement Learningcited by 15
  11. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  12. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  13. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 14
  14. 2025Efficient Robotic Policy Learning via Latent Space Backward Planningcited by 14
  15. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13
  16. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  17. 2025Dual Goal Representationscited by 13
  18. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  19. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 12
  20. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  21. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  22. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  23. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  24. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  25. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  26. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  27. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  28. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  29. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  30. 2025From Simple to Complex Skills: The Case of In-Hand Object Reorientationcited by 8
  31. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  32. 2025Local Manifold Approximation and Projection for Manifold-Aware Diffusion Planningcited by 8
  33. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  34. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  35. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  36. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  37. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  38. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7

50 most cited of 133 All 133 (JSON)