# Model-Based Offline Planning

arXiv: https://arxiv.org/abs/2008.05556 · HTML: https://mlsanity.com/paper/2008.05556/

## Key facts

- Cited by 129 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2020.

## Built on by (most cited first, 30 of 129)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners](https://mlsanity.com/paper/2302.01877/) (2023)
- [RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.12581/) (2022)
- [Offline Reinforcement Learning from Images with Latent Space Models](https://mlsanity.com/paper/2012.11547/) (2020)
- [Constraints Penalized Q-learning for Safe Offline Reinforcement Learning](https://mlsanity.com/paper/2107.09003/) (2021)
- [Online and Offline Reinforcement Learning by Planning with a Learned Model](https://mlsanity.com/paper/2104.06294/) (2021)
- [Elastic Decision Transformer](https://mlsanity.com/paper/2307.02484/) (2023)
- [DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT](https://mlsanity.com/paper/2412.19505/) (2024)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations](https://mlsanity.com/paper/2206.04779/) (2022)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning](https://mlsanity.com/paper/2202.10324/) (2022)
- [Revisiting Design Choices in Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2110.04135/) (2021)
- [Skill-based Model-based Reinforcement Learning](https://mlsanity.com/paper/2207.07560/) (2022)
- [DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learning](https://mlsanity.com/paper/2102.11492/) (2021)
- [Vector Quantized Models for Planning](https://mlsanity.com/paper/2106.04615/) (2021)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment](https://mlsanity.com/paper/2104.05632/) (2021)
- [Model-Based Runtime Monitoring with Interactive Imitation Learning](https://mlsanity.com/paper/2310.17552/) (2023)
- [Bootstrapped Transformer for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.08569/) (2022)
- [Offline Reinforcement Learning with Value-based Episodic Memory](https://mlsanity.com/paper/2110.09796/) (2021)
- [Diffusion Model Predictive Control](https://mlsanity.com/paper/2410.05364/) (2024)
- [Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2210.13846/) (2022)
- [World Models via Policy-Guided Trajectory Diffusion](https://mlsanity.com/paper/2312.08533/) (2023)
- [Residual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2106.08050/) (2021)

## Abstract (arXiv)

Offline learning is a key part of making reinforcement learning (RL) useable in real systems. Offline RL looks at scenarios where there is data from a system's operation, but no direct access to the system when learning a policy. Recent work on training RL policies from offline data has shown results both with model-free policies learned directly from the data, or with planning on top of learnt models of the data. Model-free policies tend to be more performant, but are more opaque, harder to command externally, and less easy to integrate into larger systems. We propose an offline learner that generates a model that can be used to control the system directly through planning. This allows us to have easily controllable policies directly from data, without ever interacting with the system. We show the performance of our algorithm, Model-Based Offline Planning (MBOP) on a series of robotics-inspired tasks, and demonstrate its ability leverage planning to respect environmental constraints. We are able to find near-optimal polices for certain simulated systems from as little as 50 seconds of real-time system interaction, and create zero-shot goal-conditioned policies on a series of environments. An accompanying video can be found here: https://youtu.be/nxGGHdZOFts