mlsanityData to 7 Oct 2026

Language-Guided Traffic Simulation via Scene-Level Diffusion

arXiv 2306.06344 ↗First posted 10 Jun 2023cs.RO cs.AI cs.LGcited by 110

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. However, current approaches for controlling learning-based traffic models require significant domain expertise and are difficult for practitioners to use. To remedy this, we present CTG++, a scene-level conditional diffusion model that can be guided by language instructions. Developing this requires tackling two challenges: the need for a realistic and controllable traffic model backbone, and an effective method to interface with a traffic model using language. To address these challenges, we first propose a scene-level diffusion model equipped with a spatio-temporal transformer backbone, which generates realistic and controllable traffic. We then harness a large language model (LLM) to convert a user's query into a loss function, guiding the diffusion model towards query-compliant generation. Through comprehensive evaluation, we demonstrate the effectiveness of our proposed method in generating realistic, query-compliant traffic simulations.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 28 of its 41 references that are in the map. First posted on arXiv 10 Jun 2023.

Builds on

28 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion4 Apr
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  3. 2022Guided Conditional Diffusion for Controllable Traffic Simulation31 Oct

Other references · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
13 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  3. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  4. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  5. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  6. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  7. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  8. 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
  9. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  10. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 184
  11. 2021Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Predictioncited by 131
  12. 2022BITS: Bi-level Imitation for Traffic Simulationcited by 80
  13. 2022ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planningcited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 35; 2025: 46; 2026: 18.

110 papers.

Sort:

By area:

  1. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  2. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 190
  3. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  4. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  5. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  6. 2024Large Language Models for Time Series: A Surveycited by 113
  7. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 92
  8. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  9. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 47
  10. 2023RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarioscited by 46
  11. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 41
  12. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
38 more
  1. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  2. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  3. 2023SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversariescited by 36
  4. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  5. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  6. 2024SLEDGE: Synthesizing Driving Environments with Generative Models and Rule-Based Trafficcited by 27
  7. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  8. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 26
  9. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  10. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  11. 2024Promptable Closed-loop Traffic Simulationcited by 20
  12. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
  13. 2024Probing Multimodal LLMs as World Models for Drivingcited by 19
  14. 2024Language-Driven Interactive Traffic Trajectory Generationcited by 19
  15. 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehiclescited by 18
  16. 2025Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Testcited by 18
  17. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  18. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  19. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  20. 2024FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversarialitycited by 13
  21. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 13
  22. 2025Decoupled Diffusion Sparks Adaptive Scene Generationcited by 12
  23. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  24. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  25. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 11
  26. 2025LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulationcited by 11
  27. 2025Adv-BMT: Bidirectional Motion Transformer for Safety-Critical Traffic Scenario Generationcited by 11
  28. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10
  29. 2024ScenicNL: Generating Probabilistic Scenario Programs from Natural Languagecited by 10
  30. 2024SEAL: Towards Safe Autonomous Driving via Skill-Enabled Adversary Learning for Closed-Loop Scenario Generationcited by 10
  31. 2024Words in Motion: Extracting Interpretable Control Vectors for Motion Transformerscited by 9
  32. 2024World Models: The Safety Perspectivecited by 9
  33. 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarioscited by 9
  34. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  35. 2024Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Modelscited by 8
  36. 2025Diverse Controllable Diffusion Policy with Signal Temporal Logiccited by 8
  37. 2023Prospective Role of Foundation Models in Advancing Autonomous Vehiclescited by 7
  38. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Proposal-Conditioned Latent Diffusion for Closed-Loop Traffic Scenario Generation
  2. 2025EP-Diffuser: An Efficient Diffusion Model for Traffic Scene Generation and Prediction via Polynomial Representations
  3. 2026Long-term Traffic Simulation via Structured Autoregressive Modeling
  4. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Models
  5. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion
  6. 2024Versatile Behavior Diffusion for Generalized Traffic Agent Simulation
  7. 2024Diffusion-Based Environment-Aware Trajectory Prediction
  8. 2024Data-driven Diffusion Models for Enhancing Safety in Autonomous Vehicle Traffic Simulations