mlsanityData to 14 Oct 2026

Horizon Reduction Makes RL Scalable

arXiv 2506.04168 ↗First posted 4 Jun 2025cs.LG cs.AIcited by 57

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000x larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL. Code: https://github.com/seohongpark/horizon-reduction

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 89 of its 107 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

89 of 107 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining1 Oct
  2. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  3. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models8 Feb
  4. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  5. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  6. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  7. 2023Anti-Exploration by Random Network Distillation31 Jan
  8. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  9. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  10. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
12 more
  1. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  2. 2021Offline RL Without Off-Policy Evaluation16 Jun
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  6. 2020Critic Regularized Regression26 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2022A Generalist Agentcited by 808
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  12. 2016The Option-Critic Architecturecited by 448
  13. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  14. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  15. 2019Learning Latent Plans from Playcited by 316
  16. 2024Flow Matching Guide and Codecited by 301
  17. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  18. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  19. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  20. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  21. 2022Multi-Game Decision Transformerscited by 214
  22. 2018Semi-parametric Topological Memory for Navigationcited by 209
  23. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  24. 2025Flow Q-Learningcited by 184
  25. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  26. 2019Planning with Goal-Conditioned Policiescited by 149
  27. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  28. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  29. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  30. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  31. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  32. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  33. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105
  34. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  35. 2020Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncited by 99
  36. 2024Foundation Policies with Hilbert Representationscited by 94
  37. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  38. 2024Simplifying Deep Temporal Difference Learningcited by 78

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 44.

57 papers.

Sort:

By area:

  1. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 70
  2. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
  3. 2026Q-learning with Adjoint Matchingcited by 37
  4. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  5. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  6. 2025Dual Goal Representationscited by 16
  7. 2025Value Flowscited by 15
  8. 2025The Serial Scaling Hypothesiscited by 13
  9. 2025Scalable Offline Model-Based RL with Action Chunkscited by 13
  10. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  11. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
  12. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 8
38 more
  1. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 7
  2. 2026Compositional Planning with Jumpy World Modelscited by 7
  3. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  4. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 6
  5. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  6. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 3
  7. 2026Chunk-Guided Q-Learningcited by 3
  8. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  9. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 3
  10. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  11. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 2
  12. 2026Refining Compositional Diffusion for Reliable Long-Horizon Planningcited by 2
  13. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 2
  14. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 2
  15. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learningcited by 2
  16. 2026Survival Reinforcement Learning: Toward Scalable Self-Supervised RLcited by 2
  17. 2026Reversal Q-Learningcited by 2
  18. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1
  19. 2026Action-Sufficient Goal Representationscited by 1
  20. 2026Mollified Value Learningcited by 1
  21. 2026Reward-Conditioned Reinforcement Learningcited by 1
  22. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Controlcited by 1
  23. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learningcited by 1
  24. 2026Offline RL with Hierarchical Action Chunkingcited by 1
  25. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RLcited by 1
  26. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  27. 2026Hierarchical Behaviour Spacescited by 0
  28. 2026QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RLcited by 0
  29. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  30. 2026Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planningcited by 0
  31. 2026Plan First, Diffuse Later: Extrinsic Graph Guidance for Long-Horizon Diffusion Planningcited by 0
  32. 2026Classical State Preparation for Variational Quantum Algorithms via Reinforcement Learningcited by 0
  33. 2026Goal-Conditioned Agents that Learn Everything All at Oncecited by 0
  34. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimizationcited by 0
  35. 2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
  36. 2026Dual Advantage Fieldscited by 0
  37. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  38. 2026ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learningcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Horizon Reduction as Information Loss in Offline Reinforcement Learning
  2. 2025Scaling Offline RL via Efficient and Expressive Shortcut Models
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  6. 2024Simple Ingredients for Offline Reinforcement Learning
  7. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies