mlsanityData to 7 Oct 2026

Conservative State Value Estimation for Offline Reinforcement Learning

arXiv 2302.06884 ↗First posted 14 Feb 2023cs.LG cs.AIcited by 9

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning faces a significant challenge of value over-estimation due to the distributional drift between the dataset and the current learned policy, leading to learning failure in practice. The common approach is to incorporate a penalty term to reward or value estimation in the Bellman iterations. Meanwhile, to avoid extrapolation on out-of-distribution (OOD) states and actions, existing methods focus on conservative Q-function estimation. In this paper, we propose Conservative State Value Estimation (CSVE), a new approach that learns conservative V-function via directly imposing penalty on OOD states. Compared to prior work, CSVE allows more effective state value estimation with conservative guarantees and further better policy optimization. Further, we apply CSVE and develop a practical actor-critic algorithm in which the critic does the conservative value estimation by additionally sampling and penalizing the states around the dataset, and the actor applies advantage weighted updates extended with state exploration to improve the policy. We evaluate in classic continual control tasks of D4RL, showing that our method performs better than the conservative Q-function learning methods and is strongly competitive among recent SOTA methods.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 37 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

30 of 37 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  2. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  5. 2021The Difficulty of Passive Learning in Deep Reinforcement Learning26 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2016OpenAI Gymcited by 1,684
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  10. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  11. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  12. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
1 more
  1. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency modelcited by 4
  2. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
  3. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learningcited by 1
  4. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  5. 2025Guardian: Decoupling Exploration from Safety in Reinforcement Learningcited by 0
  6. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  7. 2026When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learningcited by 0
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  9. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimates
  2. 2024Strategically Conservative Q-Learning
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2021Reducing Conservativeness Oriented Offline Reinforcement Learning
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  6. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning
  7. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
  8. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds