mlsanityData to 7 Oct 2026

MOReL : Model-Based Offline Reinforcement Learning

arXiv 2005.05951 ↗First posted 12 May 2020cs.LG cs.AI stat.MLcited by 582

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. The ability to train RL policies offline can greatly expand the applicability of RL, its data efficiency, and its experimental velocity. Prior work in offline RL has been confined almost exclusively to model-free RL approaches. In this work, we present MOReL, an algorithmic framework for model-based offline RL. This framework consists of two steps: (a) learning a pessimistic MDP (P-MDP) using the offline dataset; and (b) learning a near-optimal policy in this P-MDP. The learned P-MDP has the property that for any policy, the performance in the real environment is approximately lower-bounded by the performance in the P-MDP. This enables it to serve as a good surrogate for purposes of policy evaluation and learning, and overcome common pitfalls of model-based RL like model exploitation. Theoretically, we show that MOReL is minimax optimal (up to log factors) for offline RL. Through experiments, we show that MOReL matches or exceeds state-of-the-art results in widely studied offline RL benchmarks. Moreover, the modular design of MOReL enables future advances in its components (e.g. generative modeling, uncertainty estimation, planning etc.) to directly translate into advances for offline RL.

Key facts. Cited by 582 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 99 references that are in the map. First posted on arXiv 12 May 2020.

Builds on

55 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  3. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2016OpenAI Gymcited by 1,684
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2018Exploration by Random Network Distillationcited by 966
38 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  4. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  9. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  13. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  14. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  15. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  16. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  17. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  18. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  19. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  20. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  21. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  22. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  23. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  24. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  25. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  26. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  27. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  28. 2019Exploring Model-based Planning with Policy Networkscited by 90
  29. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  30. 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
  31. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  32. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  33. 2019Data Efficient Reinforcement Learning for Legged Robotscited by 60
  34. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  35. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  36. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  37. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  38. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 8; 2021: 92; 2022: 113; 2023: 113; 2024: 96; 2025: 84; 2026: 76.

582 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  8. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  10. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
38 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  8. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  9. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  10. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  11. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  12. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  13. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  14. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  15. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  16. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  17. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  18. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  19. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  20. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  21. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  22. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  23. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  24. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  25. 2023Synthetic Experience Replaycited by 100
  26. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  27. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  28. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  29. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  30. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  31. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  32. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  33. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  34. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  35. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  36. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  37. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  38. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72

50 most cited of 582 All 582 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Minimalist Approach to Offline Reinforcement Learning
  2. 2025Fully Offline Reinforcement Learning
  3. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  4. 2021Offline RL Without Off-Policy Evaluation
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  7. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  8. 2021The Least Restriction for Offline Reinforcement Learning