# Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models

arXiv: https://arxiv.org/abs/2507.13162 · HTML: https://mlsanity.com/paper/2507.13162/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in World models for autonomous driving, one of 11 directions in World models.
- Builds on 76 of its 89 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Placed in

- World models / [World models for autonomous driving](https://mlsanity.com/wm/world-models-for-autonomous-driving/)
- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 76)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning](https://mlsanity.com/paper/2506.09985/) (2025)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)

## Built on by (most cited first, 17 of 17)

- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [MIND: Benchmarking Memory Consistency and Action Control in World Models](https://mlsanity.com/paper/2602.08025/) (2026)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving](https://mlsanity.com/paper/2601.01528/) (2026)
- [DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning](https://mlsanity.com/paper/2604.01765/) (2026)
- [TruckDrive: Long-Range Autonomous Highway Driving Dataset](https://mlsanity.com/paper/2603.02413/) (2026)
- [CausalDrive: Real-time Causal World Models for Autonomous Driving](https://mlsanity.com/paper/2606.15341/) (2026)
- [Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset](https://mlsanity.com/paper/2603.23607/) (2026)
- [Flow Matching in Feature Space for Stochastic World Modeling](https://mlsanity.com/paper/2606.29059/) (2026)
- [What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectives](https://mlsanity.com/paper/2510.03075/) (2025)
- [Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges](https://mlsanity.com/paper/2603.09086/) (2026)
- [How Safe Will I Be Given What I Saw? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy](https://mlsanity.com/paper/2508.09346/) (2025)
- [Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning](https://mlsanity.com/paper/2606.01935/) (2026)
- [CosmosAlign: Adapting a World Foundation Model for Generative Traffic Video Forecasting](https://mlsanity.com/paper/2608.07693/) (2026)
- [LSA: Localized Semantic Alignment for Enhancing Temporal Consistency in Traffic Video Generation](https://mlsanity.com/paper/2602.05966/) (2026)
- [WorldVLM: Combining World Model Forecasting and Vision-Language Reasoning](https://mlsanity.com/paper/2603.14497/) (2026)
- [Relationally Grounded Latent World Models for Autonomous Driving](https://mlsanity.com/paper/2609.24626/) (2026)

## Abstract (arXiv)

Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios. In this work, we develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras. We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we set up a hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Code, models and qualitative results are publicly available at https://lmb-freiburg.github.io/orbis.github.io/.