mlsanityData to 14 Oct 2026

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

arXiv 1906.00949 ↗First posted 3 Jun 2019cs.LG stat.MLcited by 840

Not on the map. Why?

Cited by 593 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.

Key facts. Cited by 840 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 7; 2020: 21; 2021: 113; 2022: 137; 2023: 173; 2024: 157; 2025: 132; 2026: 100.

840 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  10. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
38 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  3. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  11. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  12. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  13. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  14. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  15. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  16. 2022Online Decision Transformercited by 196
  17. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  18. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  19. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 163
  20. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  21. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  22. 2021Offline RL Without Off-Policy Evaluationcited by 151
  23. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  24. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  25. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  26. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  27. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  28. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  29. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  30. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  31. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  32. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  33. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  34. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  35. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  36. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  37. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  38. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 114

50 most cited of 840 All 840 (JSON)