# DriveSceneGen: Generating Diverse and Realistic Driving Scenarios from Scratch

arXiv: https://arxiv.org/abs/2309.14685 · HTML: https://mlsanity.com/paper/2309.14685/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction](https://mlsanity.com/paper/2208.14437/) (2022)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [Lane Graph as Path: Continuity-preserving Path-wise Modeling for Online Lane Graph Construction](https://mlsanity.com/paper/2303.08815/) (2023)
- [HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps](https://mlsanity.com/paper/2106.14880/) (2021)

## Built on by (most cited first, 27 of 27)

- [SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout](https://mlsanity.com/paper/2412.12129/) (2024)
- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments](https://mlsanity.com/paper/2503.22496/) (2025)
- [SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Traffic](https://mlsanity.com/paper/2403.17933/) (2024)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency](https://mlsanity.com/paper/2506.07497/) (2025)
- [SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model](https://mlsanity.com/paper/2506.21976/) (2025)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [A Survey of Generative AI for Intelligent Transportation Systems: Road Transportation Perspective](https://mlsanity.com/paper/2312.08248/) (2023)
- [Generating Traffic Scenarios via In-Context Learning to Learn Better Motion Planner](https://mlsanity.com/paper/2412.18086/) (2024)
- [Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives](https://mlsanity.com/paper/2505.09074/) (2025)
- [AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Framework](https://mlsanity.com/paper/2507.13729/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [ADM: Accelerated Diffusion Model via Estimated Priors for Robust Motion Prediction under Uncertainties](https://mlsanity.com/paper/2405.00797/) (2024)
- [How Could Generative AI Support Compliance with the EU AI Act? A Review for Safe Automated Driving Perception](https://mlsanity.com/paper/2408.17222/) (2024)
- [SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction](https://mlsanity.com/paper/2506.23316/) (2025)
- [VectorWorld: Efficient Streaming World Model via Diffusion Flow on Vector Graphs](https://mlsanity.com/paper/2603.17652/) (2026)
- [PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenes](https://mlsanity.com/paper/2506.19117/) (2025)
- [DiffPlace: Street View Generation via Place-Controllable Diffusion Model Enhancing Place Recognition](https://mlsanity.com/paper/2602.11875/) (2026)
- [ControlMap: Controllable High-Definition Map Generation for Traffic Scenario Simulation](https://mlsanity.com/paper/2606.15930/) (2026)
- [Path Diffuser: Diffusion Model for Data-Driven Traffic Simulator](https://mlsanity.com/paper/2509.24995/) (2025)
- [Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Traffic](https://mlsanity.com/paper/2604.12857/) (2026)
- [ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation](https://mlsanity.com/paper/2604.17147/) (2026)
- [From Particles to Perils: SVGD-Based Hazardous Scenario Generation for Autonomous Driving Systems Testing](https://mlsanity.com/paper/2604.18918/) (2026)
- [End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation](https://mlsanity.com/paper/2607.18637/) (2026)
- [ForcingBooster: Adaptive Patch-wise Denoising for Fast Autoregressive Video Diffusion](https://mlsanity.com/paper/2610.12625/) (2026)

## Abstract (arXiv)

Realistic and diverse traffic scenarios in large quantities are crucial for the development and validation of autonomous driving systems. However, owing to numerous difficulties in the data collection process and the reliance on intensive annotations, real-world datasets lack sufficient quantity and diversity to support the increasing demand for data. This work introduces DriveSceneGen, a data-driven driving scenario generation method that learns from the real-world driving dataset and generates entire dynamic driving scenarios from scratch. DriveSceneGen is able to generate novel driving scenarios that align with real-world data distributions with high fidelity and diversity. Experimental results on 5k generated scenarios highlight the generation quality, diversity, and scalability compared to real-world datasets. To the best of our knowledge, DriveSceneGen is the first method that generates novel driving scenarios involving both static map elements and dynamic traffic participants from scratch.