# Generating Driving Scenes with Diffusion

arXiv: https://arxiv.org/abs/2305.18452 · HTML: https://mlsanity.com/paper/2305.18452/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object](https://mlsanity.com/paper/1908.05612/) (2019)
- [Arbitrary-Oriented Scene Text Detection via Rotation Proposals](https://mlsanity.com/paper/1703.01086/) (2017)
- [Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss](https://mlsanity.com/paper/2101.11952/) (2021)
- [Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergence](https://mlsanity.com/paper/2106.01883/) (2021)
- [Meta-Sim: Learning to Generate Synthetic Datasets](https://mlsanity.com/paper/1904.11621/) (2019)
- [Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions](https://mlsanity.com/paper/1906.08945/) (2019)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [What Makes for End-to-End Object Detection?](https://mlsanity.com/paper/2012.05780/) (2020)
- [Oriented Objects as pairs of Middle Lines](https://mlsanity.com/paper/1912.10694/) (2019)
- [Oriented Bounding Boxes for Small and Freely Rotated Objects](https://mlsanity.com/paper/2104.11854/) (2021)

## Built on by (most cited first, 16 of 16)

- [SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout](https://mlsanity.com/paper/2412.12129/) (2024)
- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Data-driven Traffic Simulation: A Comprehensive Review](https://mlsanity.com/paper/2310.15975/) (2023)
- [SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Models](https://mlsanity.com/paper/2311.15736/) (2023)
- [SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model](https://mlsanity.com/paper/2506.21976/) (2025)
- [Applications of Large Scale Foundation Models for Autonomous Driving](https://mlsanity.com/paper/2311.12144/) (2023)
- [Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey](https://mlsanity.com/paper/2411.02914/) (2024)
- [Reliable Trajectory Prediction and Uncertainty Quantification with Conditioned Diffusion Models](https://mlsanity.com/paper/2405.14384/) (2024)
- [UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarios](https://mlsanity.com/paper/2405.03807/) (2024)
- [Prospective Role of Foundation Models in Advancing Autonomous Vehicles](https://mlsanity.com/paper/2405.02288/) (2023)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Data-driven Diffusion Models for Enhancing Safety in Autonomous Vehicle Traffic Simulations](https://mlsanity.com/paper/2410.04809/) (2024)
- [Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation](https://mlsanity.com/paper/2502.12178/) (2025)
- [Causal-Entity Reflected Egocentric Traffic Accident Video Synthesis](https://mlsanity.com/paper/2506.23263/) (2025)
- [TSDiT: Traffic Scene Diffusion Models With Transformers](https://mlsanity.com/paper/2405.02289/) (2023)

## Abstract (arXiv)

In this paper we describe a learned method of traffic scene generation designed to simulate the output of the perception system of a self-driving car. In our "Scene Diffusion" system, inspired by latent diffusion, we use a novel combination of diffusion and object detection to directly create realistic and physically plausible arrangements of discrete bounding boxes for agents. We show that our scene generation model is able to adapt to different regions in the US, producing scenarios that capture the intricacies of each region.