mlsanityData to 7 Oct 2026

A Survey on Offline Model-Based Reinforcement Learning

arXiv 2305.03360 ↗First posted 5 May 2023cs.LG cs.AI cs.SYcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based approaches are becoming increasingly popular in the field of offline reinforcement learning, with high potential in real-world applications due to the model's capability of thoroughly utilizing the large historical datasets available with supervised learning techniques. This paper presents a literature review of recent work in offline model-based reinforcement learning, a field that utilizes model-based approaches in offline reinforcement learning. The survey provides a brief overview of the concepts and recent developments in both offline reinforcement learning and model-based reinforcement learning, and discuss the intersection of the two fields. We then presents key relevant papers in the field of offline model-based reinforcement learning and discuss their methods, particularly their approaches in solving the issue of distributional shift, the main problem faced by all current offline model-based reinforcement learning methods. We further discuss key challenges faced by the field, and suggest possible directions for future work.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 19 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

17 of 19 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  3. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 5

  1. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  2. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  3. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  4. 2022A Survey on Model-based Reinforcement Learningcited by 65
  5. 2022BcMON: Blockchain Middleware for Offline Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  2. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 18
  3. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  4. 2025Imagined Autocurriculacited by 4
  5. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  6. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  7. 2024Facilitating Reinforcement Learning for Process Control Using Transfer Learning: Overview and Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning
  4. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  5. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  6. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  7. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
  8. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective