# DITTO: Offline Imitation Learning with World Models

arXiv: https://arxiv.org/abs/2302.03086 · HTML: https://mlsanity.com/paper/2302.03086/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 6 Feb 2023.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.12581/) (2022)
- [Combining Deep Reinforcement Learning and Search for Imperfect-Information Games](https://mlsanity.com/paper/2007.13544/) (2020)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities](https://mlsanity.com/paper/1803.03453/) (2018)
- [MobILE: Model-Based Imitation Learning From Observation Alone](https://mlsanity.com/paper/2102.10769/) (2021)
- [Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation](https://mlsanity.com/paper/1905.06750/) (2019)
- [Visual Adversarial Imitation Learning using Variational Models](https://mlsanity.com/paper/2107.08829/) (2021)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Imitation Learning by Reinforcement Learning](https://mlsanity.com/paper/2108.04763/) (2021)
- [Learning Interactive Driving Policies via Data-driven Simulation](https://mlsanity.com/paper/2111.12137/) (2021)
- [Benchmarking End-to-End Behavioural Cloning on Video Games](https://mlsanity.com/paper/2004.00981/) (2020)

## Built on by (most cited first, 21 of 21)

- [Model-Based Runtime Monitoring with Interactive Imitation Learning](https://mlsanity.com/paper/2310.17552/) (2023)
- [DiWA: Diffusion Policy Adaptation with World Models](https://mlsanity.com/paper/2508.03645/) (2025)
- [SEABO: A Simple Search-Based Method for Offline Imitation Learning](https://mlsanity.com/paper/2402.03807/) (2024)
- [LUMOS: Language-Conditioned Imitation Learning with World Models](https://mlsanity.com/paper/2503.10370/) (2025)
- [CEIL: Generalized Contextual Imitation Learning](https://mlsanity.com/paper/2306.14534/) (2023)
- [Efficient Imitation Learning with Conservative World Models](https://mlsanity.com/paper/2405.13193/) (2024)
- [A Dual Approach to Imitation Learning from Observations with Offline Datasets](https://mlsanity.com/paper/2406.08805/) (2024)
- [Language-Conditioned World Modeling for Visual Navigation](https://mlsanity.com/paper/2603.26741/) (2026)
- [Reward-free World Models for Online Imitation Learning](https://mlsanity.com/paper/2410.14081/) (2024)
- [Enhancing Policy Learning with World-Action Model](https://mlsanity.com/paper/2603.28955/) (2026)
- [Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient](https://mlsanity.com/paper/2410.08893/) (2024)
- [Context-Former: Stitching via Latent Conditioned Sequence Modeling](https://mlsanity.com/paper/2401.16452/) (2024)
- [Approximate Shielding of Atari Agents for Safe Exploration](https://mlsanity.com/paper/2304.11104/) (2023)
- [Overcoming Knowledge Barriers: Online Imitation Learning from Visual Observation with Pretrained World Models](https://mlsanity.com/paper/2404.18896/) (2024)
- [Visual Imitation Learning of Non-Prehensile Manipulation Tasks with Dynamics-Supervised Models](https://mlsanity.com/paper/2410.19379/) (2024)
- [Learning from Demonstrations via Capability-Aware Goal Sampling](https://mlsanity.com/paper/2601.08731/) (2026)
- [Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning](https://mlsanity.com/paper/2505.02228/) (2025)
- [Tackling GNARLy Problems: Graph Neural Algorithmic Reasoning Reimagined through Reinforcement Learning](https://mlsanity.com/paper/2509.18930/) (2025)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [Beyond Conservatism: Recoverability-Conditioned Exploration for Model-Based Imitation Learning](https://mlsanity.com/paper/2609.33336/) (2026)
- [Dream to Remember: Continual Steering of Generalist Robot Policies with World Models](https://mlsanity.com/paper/2610.13816/) (2026)

## Abstract (arXiv)

For imitation learning algorithms to scale to real-world challenges, they must handle high-dimensional observations, offline learning, and policy-induced covariate-shift. We propose DITTO, an offline imitation learning algorithm which addresses all three of these problems. DITTO optimizes a novel distance metric in the latent space of a learned world model: First, we train a world model on all available trajectory data, then, the imitation agent is unrolled from expert start states in the learned model, and penalized for its latent divergence from the expert dataset over multiple time steps. We optimize this multi-step latent divergence using standard reinforcement learning algorithms, which provably induces imitation learning, and empirically achieves state-of-the art performance and sample efficiency on a range of Atari environments from pixels, without any online environment access. We also adapt other standard imitation learning algorithms to the world model setting, and show that this considerably improves their performance. Our results show how creative use of world models can lead to a simple, robust, and highly-performant policy-learning framework.