mlsanityData to 14 Oct 2026

Behavior Regularized Offline Reinforcement Learning

arXiv 1911.11361 ↗First posted 26 Nov 2019cs.LG cs.AI stat.MLcited by 662

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

Key facts. Cited by 662 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 37 references that are in the map. First posted on arXiv 26 Nov 2019.

Builds on

21 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2017Improved Training of Wasserstein GANscited by 2,235
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  11. 2018Maximum a Posteriori Policy Optimisationcited by 328
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
8 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  3. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  4. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  5. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  6. 2019Domain Adaptation with Asymmetrically-Relaxed Distribution Alignmentcited by 61
  7. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  8. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 17; 2021: 93; 2022: 118; 2023: 131; 2024: 103; 2025: 100; 2026: 100.

662 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
38 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  6. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  8. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  9. 2025Flow Q-Learningcited by 184
  10. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  11. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  12. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  13. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  14. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  15. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  16. 2021Offline RL Without Off-Policy Evaluationcited by 151
  17. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 149
  18. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  19. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  20. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  21. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  22. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  23. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  24. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  25. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  26. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  27. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  28. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  29. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  30. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  31. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  32. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  33. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  34. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  35. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  36. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  37. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  38. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98

50 most cited of 662 All 662 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Critic Regularized Regression
  2. 2022Model-Based Offline Meta-Reinforcement Learning with Regularization
  3. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  4. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
  5. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  6. 2024Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning
  7. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  8. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretraining