mlsanityData to 7 Oct 2026

Continuous Doubly Constrained Batch Reinforcement Learning

arXiv 2102.09225 ↗First posted 18 Feb 2021cs.LG stat.MLcited by 27

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of 32 continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 80 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

35 of 80 references.

Closest ancestors · 11

Same direction, newest first

  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  12. 2018An Algorithmic Perspective on Imitation Learningcited by 455
12 more
  1. 2017Deep Reinforcement Learning: An Overviewcited by 327
  2. 2019Challenges of Real-World Reinforcement Learningcited by 282
  3. 2017Count-Based Exploration with Neural Density Modelscited by 255
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  6. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  7. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  8. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  9. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  10. 2017Deep Q-learning from Demonstrationscited by 27
  11. 2019The problem with DDPG: understanding failures in deterministic environments with sparse rewardscited by 20
  12. 2020DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 5; 2024: 6; 2025: 3; 2026: 3.

27 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  4. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  5. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  6. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  7. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  8. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  9. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  10. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  11. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
  12. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
15 more
  1. 2024AlphaRouter: Quantum Circuit Routing with Reinforcement Learning and Tree Searchcited by 8
  2. 2024Grounded Answers for Multi-agent Decision-making Problem through Generative World Modelcited by 6
  3. 2022Time-Varying Propensity Score to Bridge the Gap between the Past and Presentcited by 5
  4. 2022State Advantage Weighting for Offline RLcited by 5
  5. 2023PASTA: Pretrained Action-State Transformer Agentscited by 5
  6. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasetscited by 4
  7. 2023Budgeting Counterfactual for Offline RLcited by 4
  8. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  9. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  10. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learningcited by 3
  11. 2025Towards Optimal Offline Reinforcement Learningcited by 2
  12. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  13. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Trainingcited by 0
  14. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  15. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Provably Good Batch Reinforcement Learning Without Great Exploration
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  3. 2021A Minimalist Approach to Offline Reinforcement Learning
  4. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  5. 2024State-Constrained Offline Reinforcement Learning
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2021The Least Restriction for Offline Reinforcement Learning