# Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments

arXiv: https://arxiv.org/abs/2503.22496 · HTML: https://mlsanity.com/paper/2503.22496/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 76 of its 95 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 30 of 76)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Equivariant Diffusion for Molecule Generation in 3D](https://mlsanity.com/paper/2203.17003/) (2022)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability](https://mlsanity.com/paper/2405.17398/) (2024)
- [MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2208.14437/) (2022)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [Neural Scene Graphs for Dynamic Scenes](https://mlsanity.com/paper/2011.10379/) (2020)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision](https://mlsanity.com/paper/2311.02077/) (2023)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)

## Built on by (most cited first, 28 of 28)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Decoupled Diffusion Sparks Adaptive Scene Generation](https://mlsanity.com/paper/2504.10485/) (2025)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustness](https://mlsanity.com/paper/2601.12672/) (2026)
- [Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happen](https://mlsanity.com/paper/2505.21743/) (2025)
- [Scaling Self-Play for End-to-End Driving](https://mlsanity.com/paper/2606.19641/) (2026)
- [Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation](https://mlsanity.com/paper/2506.17213/) (2025)
- [SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction](https://mlsanity.com/paper/2506.23316/) (2025)
- [Optimization-Guided Diffusion for Interactive Scene Generation](https://mlsanity.com/paper/2512.07661/) (2025)
- [VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs](https://mlsanity.com/paper/2603.17652/) (2026)
- [ReactSim-Bench: Benchmarking Reactive Behavior World Model Simulation in Autonomous Driving](https://mlsanity.com/paper/2606.14058/) (2026)
- [PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenes](https://mlsanity.com/paper/2506.19117/) (2025)
- [ControlMap: Controllable High-Definition Map Generation for Traffic Scenario Simulation](https://mlsanity.com/paper/2606.15930/) (2026)
- [Scaling Curriculum Learning For Autonomous Driving](https://mlsanity.com/paper/2608.22549/) (2026)
- [Path Diffuser: Diffusion Model for Data-Driven Traffic Simulator](https://mlsanity.com/paper/2509.24995/) (2025)
- [OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models](https://mlsanity.com/paper/2603.28887/) (2026)
- [Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehicles](https://mlsanity.com/paper/2604.07378/) (2026)
- [ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation](https://mlsanity.com/paper/2604.17147/) (2026)
- [From Particles to Perils: SVGD-Based Hazardous Scenario Generation for Autonomous Driving Systems Testing](https://mlsanity.com/paper/2604.18918/) (2026)
- [FlowS: One-Step Motion Prediction via Local Transport Conditioning](https://mlsanity.com/paper/2604.26065/) (2026)
- [Guiding Neuro-Symbolic Scenario Generation with Spatio-Temporal Logic](https://mlsanity.com/paper/2605.19038/) (2026)
- [Generative Lane Topology Reasoning via Autoregressive Model with Geometry Prior](https://mlsanity.com/paper/2606.31814/) (2026)
- [ECoSim: Data Efficient Fine-Tuning for Controllable Traffic Simulation](https://mlsanity.com/paper/2607.00545/) (2026)
- [MapDreamer: Aerial Imagery Conditioned Latent Diffusion for Lane-Level Map Generation](https://mlsanity.com/paper/2607.01370/) (2026)
- [Controllable Sim Agents with Behavior Latents](https://mlsanity.com/paper/2607.02496/) (2026)
- [Agent-driven Long-tail Simulation for Autonomous Driving](https://mlsanity.com/paper/2607.04331/) (2026)
- [Generation of Vectorized Maps Beyond Vehicle View](https://mlsanity.com/paper/2609.07511/) (2026)
- [TerrainForge: Physics-Grounded Road Geometry Editing for Counterfactual Autonomous Driving](https://mlsanity.com/paper/2610.02825/) (2026)

## Abstract (arXiv)

We introduce Scenario Dreamer, a fully data-driven generative simulator for autonomous vehicle planning that generates both the initial traffic scene - comprising a lane graph and agent bounding boxes - and closed-loop agent behaviours. Existing methods for generating driving simulation environments encode the initial traffic scene as a rasterized image and, as such, require parameter-heavy networks that perform unnecessary computation due to many empty pixels in the rasterized scene. Moreover, we find that existing methods that employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer instead employs a novel vectorized latent diffusion model for initial scene generation that directly operates on the vectorized scene elements and an autoregressive Transformer for data-driven agent behaviour simulation. Scenario Dreamer additionally supports scene extrapolation via diffusion inpainting, enabling the generation of unbounded simulation environments. Extensive experiments show that Scenario Dreamer outperforms existing generative simulators in realism and efficiency: the vectorized scene-generation base model achieves superior generation quality with around 2x fewer parameters, 6x lower generation latency, and 10x fewer GPU training hours compared to the strongest baseline. We confirm its practical utility by showing that reinforcement learning planning agents are more challenged in Scenario Dreamer environments than traditional non-generative simulation environments, especially on long and adversarial driving environments.