mlsanityData to 7 Oct 2026

COMBO: Conservative Offline Model-Based Policy Optimization

arXiv 2102.08363 ↗First posted 16 Feb 2021cs.LG cs.AI cs.ROcited by 365

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

Key facts. Cited by 365 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 74 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

45 of 74 references.

Closest ancestors · 13

Same direction, newest first

  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  3. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  4. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  5. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  6. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018DeepMind Control Suitecited by 935
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
20 more
  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  4. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  5. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  8. 2020Model-Based Offline Planningcited by 126
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  11. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  12. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  13. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  14. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  15. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  16. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  17. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  18. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  19. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  20. 2018Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 77; 2023: 90; 2024: 85; 2025: 53; 2026: 37.

365 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  3. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  9. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  10. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  12. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
38 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  3. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  4. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  5. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  9. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  10. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  11. 2022A Survey on Model-based Reinforcement Learningcited by 65
  12. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  13. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  14. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  15. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  16. 2025Horizon Reduction Makes RL Scalablecited by 55
  17. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  18. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  19. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  20. 2021Offline Reinforcement Learning with Reverse Model-based Imaginationcited by 47
  21. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  22. 2023A Survey on Causal Reinforcement Learningcited by 43
  23. 2023Behavior Proximal Policy Optimizationcited by 43
  24. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  25. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  26. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  27. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 40
  28. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  29. 2022Skill-based Meta-Reinforcement Learningcited by 36
  30. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  31. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  32. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  33. 2024Diffusion Model Predictive Controlcited by 35
  34. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  35. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  36. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
  37. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  38. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31

50 most cited of 365 All 365 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  2. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  4. 2025Fully Offline Reinforcement Learning
  5. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
  6. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  7. 2024Offline Trajectory Optimization for Offline Reinforcement Learning
  8. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol