mlsanityData to 7 Oct 2026

Regularized Behavior Value Estimation

arXiv 2103.09575 ↗First posted 17 Mar 2021cs.LGcited by 32

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by the overestimation of values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time. Further, R-BVE uses a ranking regularisation term that favours actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from bsuite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 53 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

30 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  2. 2020Critic Regularized Regression26 Jun
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  10. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct
  11. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  11. 2018Learning Dexterous In-Hand Manipulationcited by 643
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
7 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  2. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  3. 2019Behaviour Suite for Reinforcement Learningcited by 129
  4. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  5. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  6. 2019Ranking Policy Gradientcited by 4
  7. 2020ConQUR: Mitigating Delusional Bias in Deep Q-learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 7; 2024: 3; 2025: 3; 2026: 1.

32 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2021Offline RL Without Off-Policy Evaluationcited by 150
  4. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  5. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  6. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  7. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  8. 2021Heuristic-Guided Reinforcement Learningcited by 41
  9. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  10. 2021Active Offline Policy Selectioncited by 26
  11. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  12. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
20 more
  1. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  2. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  3. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  4. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  5. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  6. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 8
  7. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  8. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learningcited by 4
  9. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  10. 2022Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learningcited by 2
  11. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Samplingcited by 2
  12. 2023Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learningcited by 2
  13. 2025FairDICE: Fairness-Driven Offline Multi-Objective Reinforcement Learningcited by 2
  14. 2023Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learningcited by 1
  15. 2024Integrating Domain Knowledge for handling Limited Data in Offline RLcited by 1
  16. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  17. 2025SEG-Parking: Towards Safe, Efficient, and Generalizable Autonomous Parking via End-to-End Offline Reinforcement Learningcited by 1
  18. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RLcited by 1
  19. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimatescited by 0
  20. 2023Hierarchical Reinforcement Learning in Complex 3D Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2022Behavior Estimation from Multi-Source Data for Offline Reinforcement Learning
  5. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  6. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  7. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  8. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL