mlsanityData to 7 Oct 2026

Horizon Adaptive Offline Policy Learning via Value Stitching

arXiv 2606.21136 ↗First posted 19 Jun 2026cs.LG cs.ROcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 85 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

58 of 85 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Reinforcement Learning with Action Chunking10 Jul
  2. 2025Horizon Reduction Makes RL Scalable4 Jun
  3. 2025Scaling Offline RL via Efficient and Expressive Shortcut Models28 May
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  5. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models8 Feb
  6. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL7 Jun
  7. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  8. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  9. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  10. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  11. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
4 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 42

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  11. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  12. 2016The Option-Critic Architecturecited by 433
30 more
  1. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  3. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  5. 2025Flow Q-Learningcited by 162
  6. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  8. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  9. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  10. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  11. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  12. 2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
  13. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  14. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  15. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  16. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  17. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  18. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  19. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  20. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  21. 2026Q-learning with Adjoint Matchingcited by 33
  22. 2018Learning Abstract Optionscited by 31
  23. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29
  24. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  25. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  26. 2025Value Flowscited by 12
  27. 2026Unleashing the Potential of Diffusion Models for End-to-End Autonomous Drivingcited by 11
  28. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
  29. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  30. 2025Dichotomous Diffusion Policy Optimizationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  2. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
  3. 2026Offline Reinforcement Learning with Universal Horizon Models
  4. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  5. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
  6. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
  7. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  8. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization