mlsanityData to 7 Oct 2026

Upper-Expectile Multi-Step Q-Learning for Off-Policy Reinforcement Learning

arXiv 2608.02034 ↗First posted 3 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-step returns accelerate reward propagation in off-policy reinforcement learning, but couple the evaluation of each decision to the suboptimal logged actions that follow it, inducing a pessimistic bias that grows with the horizon. We propose Expectile n-step Q-learning (ENQ), which replaces the symmetric n-step temporal-difference (TD) loss with an asymmetric expectile loss on the action-value error, with expectile level τ as the only method-specific hyperparameter added beyond n-step TD. We prove that the ENQ operator is a γn-contraction. Under deterministic dynamics, at τ=1, its bias vanishes at the optimal action-value function Q^* on covered in-support pairs, and the corresponding fixed point satisfies the separation-n instance and its multiples of the lower-bound inequality used by Long-Horizon Q-learning (LQL). Under stochastic dynamics, the operator bias admits two-sided bounds with horizon-independent noise constants. Using a single expectile level τ=0.8 and a fixed backup horizon across 27 manipulation and navigation task instances, ENQ is competitive with LQL on aggregate, achieves higher measured training-step throughput in our profiling study, and benefits more from a ten-critic ensemble in a controlled scaling experiment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

36 of 50 references.

References · 36

  1. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  12. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
24 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  3. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2020Revisiting Fundamentals of Experience Replaycited by 171
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  10. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  11. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  12. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  13. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  14. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  15. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
  16. 2023Anti-Exploration by Random Network Distillationcited by 49
  17. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  18. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  19. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  20. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  21. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learningcited by 17
  22. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  23. 2025Value-Based Deep RL Scales Predictablycited by 14
  24. 2024Averaging n-step Returns Reduces Variance in Reinforcement Learningcited by 9

Built on by

Not cited yet.