mlsanityData to 13 Oct 2026

Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

arXiv 2006.03647 ↗First posted 5 Jun 2020cs.LG cs.AI stat.MLcited by 120

Not on the map. Why?

Cited by 75 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most reinforcement learning (RL) algorithms assume online access to the environment, in which one may readily interleave updates to the policy with experience collection using that policy. However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploying a new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than a few times during learning. With this view, we propose a novel concept of deployment efficiency, measuring the number of distinct data-collection policies that are used during policy learning. We observe that na"ively applying existing model-free offline RL algorithms recursively does not lead to a practical deployment-efficient and sample-efficient algorithm. We propose a novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN) that can effectively optimize a policy offline using 10-20 times fewer data than prior works. Furthermore, the recursive application of BREMEN is able to achieve impressive deployment efficiency while maintaining the same or better sample efficiency, learning successful policies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds to millions in standard RL baselines. Codes and pre-trained models are available at https://github.com/matsuolab/BREMEN .

Key facts. Cited by 120 papers in the map (data through 13 Oct 2026). First posted on arXiv 5 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 2; 2021: 25; 2022: 27; 2023: 21; 2024: 26; 2025: 12; 2026: 7.

120 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 884
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 366
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  6. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 165
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 161
  8. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  10. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 128
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  12. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
38 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  3. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  5. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  6. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  9. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  10. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  11. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  12. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 46
  13. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  14. 2025What Makes a Good Diffusion Planner for Decision Making?cited by 37
  15. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 36
  16. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  17. 2022Sample-Efficient Reinforcement Learning with loglog(T) Switching Costcited by 32
  18. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  19. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 27
  20. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  21. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RLcited by 24
  22. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  23. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  24. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  25. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  26. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
  27. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 22
  28. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  29. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  30. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  31. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  32. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 20
  33. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 19
  34. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  35. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  36. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  37. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  38. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020MOPO: Model-based Offline Policy Optimization
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2020MOReL : Model-Based Offline Reinforcement Learning
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  5. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  6. 2023Deploying Offline Reinforcement Learning with Human Feedback
  7. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  8. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance