mlsanityData to 7 Oct 2026

Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning

arXiv 2605.14779 ↗First posted 14 May 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q(λ) (CPQL). Our algorithm adapts the Peng's Q(λ) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is the first work in offline RL to theoretically and empirically demonstrate the effectiveness of conservative value estimation with a textitmulti-step operator by fully leveraging offline trajectories. The fixed point of the PQL operator in offline RL lies closer to the value function of the behavior policy, thereby naturally inducing implicit behavior regularization. CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees -- a milestone that previous conservative approaches could not achieve. Extensive numerical experiments on the D4RL benchmark demonstrate that CPQL consistently and significantly outperforms existing offline single-step baselines. In addition to the contributions of CPQL in offline RL, our proposed method also contributes to the offline-to-online learning framework. Using the Q-function pre-trained by CPQL in offline settings enables the online PQL agent to avoid the performance drop typically observed at the start of fine-tuning and to attain robust performance improvements. Our code is available at https://github.com/oh-lab/CPQL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 72 of its 86 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

72 of 86 references.

Closest ancestors · 47

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model27 Oct
  3. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  4. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
  5. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning23 May
  6. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization6 Nov
  7. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning22 Sep
  8. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
  9. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  10. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  12. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
35 more
  1. 2023Behavior Proximal Policy Optimization22 Feb
  2. 2023Conservative State Value Estimation for Offline Reinforcement Learning14 Feb
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2023Anti-Exploration by Random Network Distillation31 Jan
  5. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  6. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  7. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  8. 2022Mutual Information Regularized Offline Reinforcement Learning14 Oct
  9. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  10. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward13 Jun
  11. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  12. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  13. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  14. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  15. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  16. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  17. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  18. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  19. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  20. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  21. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  23. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  24. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  25. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  26. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  27. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  28. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  29. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  30. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  31. 2020MOPO: Model-based Offline Policy Optimization27 May
  32. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  33. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  34. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  35. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
13 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  4. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  5. 2022Jump-Start Reinforcement Learningcited by 102
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  7. 2016Q(λ) with Off-Policy Correctionscited by 48
  8. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  9. 2019Adaptive Trade-Offs in Off-Policy Learningcited by 19
  10. 2021Revisiting Peng's Q(λ) for Modern Reinforcement Learningcited by 19
  11. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  12. 2017Investigating Recurrence and Eligibility Traces in Deep Q-Networkscited by 8
  13. 2023Trajectory-Aware Eligibility Traces for Off-Policy Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  4. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  5. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  6. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  7. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimates
  8. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds