# Separating the World and Ego Models for Self-Driving

arXiv: https://arxiv.org/abs/2204.07184 · HTML: https://mlsanity.com/paper/2204.07184/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2022.

## Built on by (most cited first, 4 of 4)

- [Model-Based Imitation Learning for Urban Driving](https://mlsanity.com/paper/2210.07729/) (2022)
- [Learning Interaction-aware Motion Prediction Model for Decision-making in Autonomous Driving](https://mlsanity.com/paper/2302.03939/) (2023)
- [Self-Supervised JEPA-based World Models for LiDAR Occupancy Completion and Forecasting](https://mlsanity.com/paper/2602.12540/) (2026)
- [Dual policy as self-model for planning](https://mlsanity.com/paper/2306.04440/) (2023)

## Abstract (arXiv)

Training self-driving systems to be robust to the long-tail of driving scenarios is a critical problem. Model-based approaches leverage simulation to emulate a wide range of scenarios without putting users at risk in the real world. One promising path to faithful simulation is to train a forward model of the world to predict the future states of both the environment and the ego-vehicle given past states and a sequence of actions. In this paper, we argue that it is beneficial to model the state of the ego-vehicle, which often has simple, predictable and deterministic behavior, separately from the rest of the environment, which is much more complex and highly multimodal. We propose to model the ego-vehicle using a simple and differentiable kinematic model, while training a stochastic convolutional forward model on raster representations of the state to predict the behavior of the rest of the environment. We explore several configurations of such decoupled models, and evaluate their performance both with Model Predictive Control (MPC) and direct policy learning. We test our methods on the task of highway driving and demonstrate lower crash rates and better stability. The code is available at https://github.com/vladisai/pytorch-PPUU/tree/ICLR2022.