mlsanityData to 14 Oct 2026

DCE: Offline Reinforcement Learning With Double Conservative Estimates

arXiv 2209.13132 ↗First posted 27 Sep 2022cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning has attracted much interest in solving the application challenge for traditional reinforcement learning. Offline reinforcement learning uses previously-collected datasets to train agents without any interaction. For addressing the overestimation of OOD (out-of-distribution) actions, conservative estimates give a low value for all inputs. Previous conservative estimation methods are usually difficult to avoid the impact of OOD actions on Q-value estimates. In addition, these algorithms usually need to lose some computational efficiency to achieve the purpose of conservative estimation. In this paper, we propose a simple conservative estimation method, double conservative estimates (DCE), which use two conservative estimation method to constraint policy. Our algorithm introduces V-function to avoid the error of in-distribution action while implicit achieving conservative estimation. In addition, our algorithm uses a controllable penalty term changing the degree of conservatism in training. We theoretically show how this method influences the estimation of OOD actions and in-distribution actions. Our experiment separately shows that two conservative estimation methods impact the estimation of all state-action. DCE demonstrates the state-of-the-art performance on D4RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 31 references that are in the map. First posted on arXiv 27 Sep 2022.

Builds on

28 of 31 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  3. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  6. 2021Offline RL Without Off-Policy Evaluation16 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Regularized Behavior Value Estimation17 Mar
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
9 more
  1. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs18 Oct
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Critic Regularized Regression26 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 7

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  7. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  2. 2024Learning from Sparse Offline Datasets via Conservative Density Estimation
  3. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  4. 2024Strategically Conservative Q-Learning
  5. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  6. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
  7. 2024CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning