mlsanityData to 14 Oct 2026

Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities

arXiv 2605.05812 ↗First posted 7 May 2026cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 52 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

34 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Averaging n-step Returns Reduces Variance in Reinforcement Learning6 Feb

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  10. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
21 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  5. 2025π*0.6: a VLA That Learns From Experiencecited by 303
  6. 2024Diffusion Policy Policy Optimizationcited by 280
  7. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  8. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  9. 2025Flow Q-Learningcited by 184
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  11. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  12. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  13. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  14. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  15. 2025Reinforcement Learning with Action Chunkingcited by 107
  16. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  17. 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
  18. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
  19. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  20. 2025Joint Distillation for Fast Likelihood Evaluation and Sampling in Flow-based Modelscited by 18
  21. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Is Q-learning an Ill-posed Problem?
  2. 2024Time-Scale Separation in Q-Learning: Extending TD(triangle) for Action-Value Function Decomposition
  3. 2026Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning
  4. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
  5. 2026Horizon Adaptive Offline Policy Learning via Value Stitching
  6. 2025An Analysis of Action-Value Temporal-Difference Methods That Learn State Values
  7. 2022Learning Value Functions from Undirected State-only Experience
  8. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression