mlsanityData to 14 Oct 2026

Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning

arXiv 2508.05960 ↗First posted 8 Aug 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without further environment interaction. A key challenge is the distribution shift between the learned and behavior policies, leading to out-of-distribution (OOD) actions and overestimation. To prevent gross overestimation, the value function must remain conservative; however, excessive conservatism may hinder performance improvement. To address this, we propose the mildly conservative regularized evaluation (MCRE) framework, which balances conservatism and performance by combining temporal difference (TD) error with a behavior cloning term in the Bellman backup. Building on this, we develop the mildly conservative regularized Q-learning (MCRQ) algorithm, which integrates MCRE into an off-policy actor-critic framework. Experiments show that MCRQ outperforms strong baselines and state-of-the-art offline RL algorithms on benchmark datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 40 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

29 of 40 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning10 Jul
  2. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching4 Feb
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update1 Feb
  4. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance4 Sep
  5. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  6. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  8. 2023Conservative State Value Estimation for Offline Reinforcement Learning14 Feb
  9. 2022Mutual Information Regularized Offline Reinforcement Learning14 Oct
  10. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
8 more
  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 9

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2016OpenAI Gymcited by 1,736
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  7. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  8. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 26
  9. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  4. 2024Strategically Conservative Q-Learning
  5. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
  6. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
  7. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  8. 2021Reducing Conservativeness Oriented Offline Reinforcement Learning