mlsanityData to 14 Oct 2026

Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion

arXiv 2311.02738 ↗First posted 5 Nov 2023cs.LG cs.CV cs.ROcited by 47

Not on the map. Why?

Abstract (arXiv)

Automated creation of synthetic traffic scenarios is a key part of validating the safety of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. To provide additional control over the generated scenario, this distribution is conditioned on a map and sets of tokens describing the desired scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns and generalizes to different geographical regions.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 5 Nov 2023.

Builds on

33 of 40 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  10. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
21 more
  1. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  2. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  3. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  4. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 193
  5. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 190
  6. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  7. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
  8. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  9. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  10. 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 106
  11. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
  12. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 96
  13. 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 84
  14. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  15. 2021SimNet: Learning Reactive Self-driving Simulations from Real-world Observationscited by 77
  16. 2020Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Methodcited by 75
  17. 2022Symphony: Learning Realistic and Diverse Agents for Autonomous Driving Simulationcited by 61
  18. 2020Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Datacited by 58
  19. 2020What Makes for End-to-End Object Detection?cited by 49
  20. 2022TrajGen: Generating Realistic and Diverse Trajectories with Reactive and Feasible Agent Behaviors for Autonomous Drivingcited by 23
  21. 2022LatentFormer: Multi-Agent Transformer-Based Interaction Modeling and Trajectory Predictioncited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 23; 2026: 12.

47 papers.

Sort:

By area:

  1. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  2. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 67
  3. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 48
  4. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  5. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  6. 2023SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversariescited by 38
  7. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  8. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 28
  9. 2024SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Trafficcited by 27
  10. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  11. 2025Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistencycited by 22
  12. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
35 more
  1. 2024Diffusion-Based Environment-Aware Trajectory Predictioncited by 14
  2. 2025Adv-BMT: Bidirectional Motion Transformer for Safety-Critical Traffic Scenario Generationcited by 11
  3. 2024ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stablecited by 8
  4. 2025Continuous Locomotive Crowd Behavior Generationcited by 7
  5. 2025Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Modelscited by 7
  6. 2024Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Surveycited by 6
  7. 2025Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Modelcited by 6
  8. 2024SUSTechGAN: Image Generation for Object Detection in Adverse Conditions of Autonomous Drivingcited by 5
  9. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Frameworkcited by 5
  10. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 5
  11. 2025Safety-Critical Traffic Simulation with Guided Latent Diffusion Modelcited by 4
  12. 2025Generative Data Mining with Longtail-Guided Diffusioncited by 3
  13. 2025V-Max: A Reinforcement Learning Framework for Autonomous Drivingcited by 3
  14. 2025SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integrationcited by 3
  15. 2025Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generationcited by 2
  16. 2025Controllable Motion Generation via Diffusion Modal Couplingcited by 2
  17. 2025Bounding Box-Guided Diffusion for Synthesizing Industrial Images and Segmentation Mapcited by 2
  18. 2025SceneStreamer: Continuous Scenario Generation as Next Token Group Predictioncited by 2
  19. 2026AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Modelscited by 2
  20. 2026Net-Ev2: A Generative Simulator for Network Event Evolutioncited by 1
  21. 2024Foundation Models for Rapid Autonomy Validationcited by 0
  22. 2025Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictionscited by 0
  23. 2025Linear Memory SE(2) Invariant Attentioncited by 0
  24. 2025Path Diffuser: Diffusion Model for Data-Driven Traffic Simulatorcited by 0
  25. 2025From Camera-Based Sensing to Reasoning: A Comprehensive Review Toward Proactive Vulnerable Road User Safetycited by 0
  26. 2025Multi-Agent Scenario Generation in Roundabouts with a Transformer-enhanced Conditional Variational Autoencodercited by 0
  27. 2026Emergency Lane-Change Simulation: A Behavioral Guidance Approach for Risky Scenario Generationcited by 0
  28. 2026Evaluation as Evolution: Transforming Adversarial Diffusion into Closed-Loop Curricula for Autonomous Vehiclescited by 0
  29. 2026Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Trafficcited by 0
  30. 2026ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generationcited by 0
  31. 2026Guiding Neuro-Symbolic Scenario Generation with Spatio-Temporal Logiccited by 0
  32. 2026TrafficAlign: Aligning Large Language Models for Traffic Scenario Generationcited by 0
  33. 2026Controllable Sim Agents with Behavior Latentscited by 0
  34. 2026End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generationcited by 0
  35. 2026Controllable Road Marking Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Controllable Latent Diffusion for Traffic Simulation
  2. 2026Proposal-Conditioned Latent Diffusion for Closed-Loop Traffic Scenario Generation
  3. 2023A Diffusion-Model of Joint Interactive Navigation
  4. 2024Versatile Behavior Diffusion for Generalized Traffic Agent Simulation
  5. 2025Rolling Ahead Diffusion for Traffic Scene Simulation
  6. 2024Data-driven Diffusion Models for Enhancing Safety in Autonomous Vehicle Traffic Simulations
  7. 2024DragTraffic: Interactive and Controllable Traffic Scene Generation for Autonomous Driving
  8. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior