Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Models
On the map: Retrieval-augmented generation → Multimodal RAG
also in 1 more
Abstract (arXiv)
Simulation-based testing is crucial for validating autonomous vehicles (AVs), yet existing scenario generation methods either overfit to common driving patterns or operate in an offline, non-interactive manner that fails to expose rare, safety-critical corner cases. In this paper, we introduce an online, retrieval-augmented large language model (LLM) framework for generating safety-critical driving scenarios. Our method first employs an LLM-based behavior analyzer to infer the most dangerous intent of the background vehicle from the observed state, then queries additional LLM agents to synthesize feasible adversarial trajectories. To mitigate catastrophic forgetting and accelerate adaptation, we augment the framework with a dynamic memorization and retrieval bank of intent-planner pairs, automatically expanding its behavioral library when novel intents arise. Evaluations using the Waymo Open Motion Dataset demonstrate that our model reduces the mean minimum time-to-collision from 1.62 to 1.08 s and incurs a 75% collision rate, substantially outperforming baselines.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 22 of its 28 references that are in the map. First posted on arXiv 2 May 2025.
Builds on
22 of 28 references.
Closest ancestors · 1
Same direction, newest first
Other references · 21
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
- 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 537
- 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 139
- 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 137
- 2022TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarioscited by 106
- 2022A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspectivecited by 105
- 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 93
- 2021SceneGen: Learning to Generate Realistic Traffic Scenescited by 84
- 2020Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Methodcited by 75
- 2022"Curse of rarity" for autonomous vehiclescited by 63
- 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 54
9 more
- 2020TrajGAIL: Generating Urban Vehicle Trajectories using Generative Adversarial Imitation Learningcited by 47
- 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 47
- 2023Data-driven Traffic Simulation: A Comprehensive Reviewcited by 29
- 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
- 2025LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Modelscited by 19
- 2024Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehiclescited by 18
- 2024Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priorscited by 15
- 2024GOOSE: Goal-Conditioned Reinforcement Learning for Safety-Critical Scenario Generationcited by 10
- 2024SEAL: Towards Safe Autonomous Driving via Skill-Enabled Adversary Learning for Closed-Loop Scenario Generationcited by 10
Built on by
Citing papers per year: 2025: 3; 2026: 4.
7 papers.
- 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
- 2025Steerable Adversarial Scenario Generation through Test-Time Preference Alignmentcited by 9
- 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happencited by 5
- 2026CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesiscited by 2
- 2026From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Drivingcited by 0
- 2026NSF-HRPT: Neural Semantic Field meets Hierarchical Risk Perception Tree for Safety-Critical Scenario Assessmentcited by 0
- 2026LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Realitycited by 0
- 2026NSF-HRPT: Neural Semantic Field meets Hierarchical Risk Perception Tree for Safety-Critical Scenario Assessment5 Aug 2026
- 2026LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality31 Jul 2026
- 2026CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis8 Jul 2026
- 2026From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving12 Jun 2026
- 2025Steerable Adversarial Scenario Generation through Test-Time Preference Alignment24 Sep 2025
- 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis13 Jun 2025
- 2025Simulating the Unseen: Crash Prediction Must Learn from What Did Not Happen27 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios
- 2026CrashDiffuser: VLM-Guided Collision Intent Reasoning for Fine-Grained Safety-Critical Traffic Scenario Generation
- 2026PCASim: Promptable Closed-loop Adversarial Simulation for Urban Traffic Environment ↗
- 2025Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge
- 2025LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios
- 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles
- 2025SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data ↗
- 2026SaFeR: Safety-Critical Scenario Generation for Autonomous Driving Test via Feasibility-Constrained Token Resampling