mlsanityData to 7 Oct 2026

Horizon Generalization in Reinforcement Learning

arXiv 2501.02709 ↗First posted 6 Jan 2025cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

We study goal-conditioned RL through the lens of generalization, but not in the traditional sense of random augmentations and domain randomization. Rather, we aim to learn goal-directed policies that generalize with respect to the horizon: after training to reach nearby goals (which are easy to learn), these policies should succeed in reaching distant goals (which are quite challenging to learn). In the same way that invariance is closely linked with generalization is other areas of machine learning (e.g., normalization layers make a network invariant to scale, and therefore generalize to inputs of varying scales), we show that this notion of horizon generalization is closely linked with invariance to planning: a policy navigating towards a goal will select the same actions as if it were navigating to a waypoint en route to that goal. Thus, such a policy trained to reach nearby goals should succeed at reaching arbitrarily-distant goals. Our theoretical analysis proves that both horizon generalization and planning invariance are possible, under some assumptions. We present new experimental results and recall findings from prior work in support of our theoretical results. Taken together, our results open the door to studying how techniques for invariance and generalization developed in other areas of machine learning might be adapted to achieve this alluring property.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 63 of its 89 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

63 of 89 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2016Group Equivariant Convolutional Networkscited by 1,154
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2022Large Language Models Can Self-Improvecited by 656
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  12. 2020Reinforcement Learning with Augmented Datacited by 452
38 more
  1. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  2. 2018AI safety via debatecited by 384
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  4. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  5. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  6. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  8. 2022Exploring Length Generalization in Large Language Modelscited by 204
  9. 2018Semi-parametric Topological Memory for Navigationcited by 203
  10. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  11. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  12. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  13. 2016Value Iteration Networkscited by 158
  14. 2019Unsupervised State Representation Learning in Ataricited by 144
  15. 2019Planning with Goal-Conditioned Policiescited by 144
  16. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  17. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  18. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  19. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  20. 2018Generalization and Regularization in DQNcited by 117
  21. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  22. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  23. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  24. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  25. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  26. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  27. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
  28. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  29. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  30. 2018Gotta Learn Fast: A New Benchmark for Generalization in RLcited by 59
  31. 2022ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hintscited by 55
  32. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  33. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  34. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  35. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  36. 2018Gated Path Planning Networkscited by 31
  37. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28
  38. 2021Model-Based Reinforcement Learning via Latent-Space Collocationcited by 27

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  2. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  3. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  4. 2025Flattening Hierarchies with Policy Bootstrappingcited by 8
  5. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  6. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 3
  7. 2026On the Role of Computation in Reinforcement Learningcited by 3
  8. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  9. 2026Action-Sufficient Goal Representationscited by 1
  10. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  11. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 1
  12. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
3 more
  1. 2026On the Importance of Multistability for Horizon Generalization in Reinforcement Learningcited by 0
  2. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  3. 2026Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?cited by 0