mlsanityData to 7 Oct 2026

ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles

arXiv 2405.14062 ↗First posted 22 May 2024cs.AI cs.LGcited by 92

On the map: End-to-end autonomous driving → Large language models for driving

Abstract (arXiv)

We present ChatScene, a Large Language Model (LLM)-based agent that leverages the capabilities of LLMs to generate safety-critical scenarios for autonomous vehicles. Given unstructured language instructions, the agent first generates textually described traffic scenarios using LLMs. These scenario descriptions are subsequently broken down into several sub-descriptions for specified details such as behaviors and locations of vehicles. The agent then distinctively transforms the textually described sub-scenarios into domain-specific languages, which then generate actual code for prediction and control in simulators, facilitating the creation of diverse and complex scenarios within the CARLA simulation environment. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets by training a knowledge database containing the scenario description and code pairs. Extensive experimental results underscore the efficacy of ChatScene in improving the safety of autonomous vehicles. For instance, the scenarios generated by ChatScene show a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, we show that by using our generated safety-critical scenarios to fine-tune different RL-based autonomous driving models, they can achieve a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Placed in Large language models for driving, one of 9 directions in End-to-end autonomous driving. Builds on 39 of its 59 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

39 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model2 Oct
  2. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Models14 Jul

Other references · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
25 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  6. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  7. 2019Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Drivingcited by 151
  8. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  9. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  10. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  11. 2018Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Datacited by 94
  12. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 89
  13. 2023Language Conditioned Traffic Generationcited by 86
  14. 2018Scenic: A Language for Scenario Specification and Scene Generationcited by 79
  15. 2020Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Methodcited by 74
  16. 2019PhysGAN: Generating Physical-World-Resilient Adversarial Examples for Autonomous Drivingcited by 72
  17. 2020SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Drivingcited by 68
  18. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 65
  19. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  20. 2020Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarioscited by 42
  21. 2017Ontology based Scene Creation for the Development of Automated Vehiclescited by 40
  22. 2019SUMMIT: A Simulator for Urban Driving in Massive Mixed Trafficcited by 39
  23. 2020Scenic: A Language for Scenario Specification and Data Generationcited by 39
  24. 2019Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Drivingcited by 36
  25. 2019CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarioscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 42; 2026: 39.

92 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  2. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  3. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  4. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 37
  5. 2024ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObilitycited by 34
  6. 2025NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Modelscited by 29
  7. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  8. 2025SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Modelscited by 22
  9. 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehiclescited by 18
  10. 2025LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Modelscited by 16
  11. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  12. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
38 more
  1. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
  2. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 11
  3. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  4. 2025UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Drivingcited by 10
  5. 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarioscited by 9
  6. 2025WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Drivingcited by 9
  7. 2025From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarioscited by 8
  8. 2026CoV: Chain-of-View Prompting for Spatial Reasoningcited by 8
  9. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  10. 2025DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Modelscited by 7
  11. 2025Tracking Meets Large Multimodal Models for Driving Scenario Understandingcited by 7
  12. 2025Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Modelscited by 7
  13. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  14. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  15. 2025SafeMVDrive: Multi-view Safety-Critical Driving Video Synthesis in the Real World Domaincited by 6
  16. 2026VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustnesscited by 6
  17. 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
  18. 2025AGENTS-LLM: Augmentative GENeration of Challenging Traffic Scenarios with an Agentic LLM Frameworkcited by 5
  19. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  20. 2024V-RoAst: Visual Road Assessment. Can VLM be a Road Safety Assessor Using the iRAP Standard?cited by 4
  21. 2024Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehiclescited by 4
  22. 2025LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systemscited by 4
  23. 2025ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detailcited by 4
  24. 2025RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulationcited by 4
  25. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehiclescited by 4
  26. 2024Adversarial and Reactive Traffic Entities for Behavior-Realistic Driving Simulation: A Reviewcited by 3
  27. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Drivingcited by 3
  28. 2025AirTrafficGen: Configurable Air Traffic Scenario Generation with Large Language Modelscited by 3
  29. 2025SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integrationcited by 3
  30. 2025AgentSUMO: An Agentic Framework for Interactive Simulation Scenario Generation in SUMO via Large Language Modelscited by 3
  31. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  32. 2026ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Drivingcited by 3
  33. 2026A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous drivingcited by 3
  34. 2024Exploring Critical Testing Scenarios for Decision-Making Policies: An LLM Approachcited by 2
  35. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  36. 2025Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Drivingcited by 2
  37. 2025LLM-based Realistic Safety-Critical Driving Video Generationcited by 2
  38. 2025LinguaSim: Interactive Multi-Vehicle Testing Scenario Generation via Natural Language Instruction Based on Large Language Modelscited by 2

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CrashAgent: Crash Scenario Generation via Multi-modal Reasoning
  2. 2025CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learning
  3. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Models
  4. 2024ScenicNL: Generating Probabilistic Scenario Programs from Natural Language
  5. 2024Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation
  6. 2026AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems
  7. 2025David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generation
  8. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integration