Data to 7 Oct 2026

MOPO: Model-based Offline Policy Optimization

arXiv 2005.13239 ↗First posted 27 May 2020cs.LG cs.AI stat.MLcited by 630

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at https://github.com/tianheyu927/mopo.

Key facts. Cited by 630 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 77 references that are in the map. First posted on arXiv 27 May 2020.

Builds on

51 of 77 references.

Closest ancestors · 5

Same direction, newest first

2020MOReL : Model-Based Offline Reinforcement Learning12 May
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
2019Behavior Regularized Offline Reinforcement Learning26 Nov
2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 46

2017Proximal Policy Optimization Algorithmscited by 14,441
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
2015Continuous control with deep reinforcement learningcited by 3,279
2015Trust Region Policy Optimizationcited by 2,503
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
2018World Modelscited by 1,444
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227

34 more

2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
2019Model-Based Reinforcement Learning for Ataricited by 495
2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
2018Quantifying Generalization in Reinforcement Learningcited by 345
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2016Deep Visual Foresight for Planning Robot Motioncited by 276
2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
2016Value Iteration Networkscited by 158
2019Benchmarking Model-Based Reinforcement Learningcited by 158
2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
2017Value Prediction Networkcited by 117
2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
2016The Predictron: End-To-End Learning and Planningcited by 92
2019Exploring Model-based Planning with Policy Networkscited by 90
2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
2020Interference and Generalization in Temporal Difference Learningcited by 41
2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
2018The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spacescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 6; 2021: 103; 2022: 123; 2023: 133; 2024: 114; 2025: 82; 2026: 69.

630 papers.

Sort:

By area:

2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
2020Is Pessimism Provably Efficient for Offline RL?cited by 362
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225

38 more

2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
2022Model-Based Imitation Learning for Urban Drivingcited by 149
2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
2022On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networkscited by 119
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
2023Synthetic Experience Replaycited by 100
2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
2022Robust Reinforcement Learning using Offline Datacited by 95
2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83

50 most cited of 630 All 630 (JSON)

Similar, not cited

Close in wording, no citation either way