# From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios

arXiv: https://arxiv.org/abs/2502.02145 · HTML: https://mlsanity.com/paper/2502.02145/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [Language Prompt for Autonomous Driving](https://mlsanity.com/paper/2309.04379/) (2023)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024)
- [ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObility](https://mlsanity.com/paper/2409.09040/) (2024)
- [Corner Case Generation and Analysis for Safety Assessment of Autonomous Vehicles](https://mlsanity.com/paper/2102.03483/) (2021)
- [Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles](https://mlsanity.com/paper/2409.06450/) (2024)
- [Robust Trajectory Prediction against Adversarial Attacks](https://mlsanity.com/paper/2208.00094/) (2022)
- [FRENETIX: A High-Performance and Modular Motion Planning Framework for Autonomous Driving](https://mlsanity.com/paper/2402.01443/) (2024)
- [Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Model](https://mlsanity.com/paper/2409.09575/) (2024)
- [Reality Bites: Assessing the Realism of Driving Scenarios with Large Language Models](https://mlsanity.com/paper/2403.09906/) (2024)
- [A new Taxonomy for Automated Driving: Structuring Applications based on their Operational Design Domain, Level of Automation and Automation Readiness](https://mlsanity.com/paper/2404.17044/) (2024)
- [DualAD: Dual-Layer Planning for Reasoning in Autonomous Driving](https://mlsanity.com/paper/2409.18053/) (2024)
- [Building Safer Autonomous Agents by Leveraging Risky Driving Behavior Knowledge](https://mlsanity.com/paper/2103.10245/) (2021)

## Built on by (most cited first, 8 of 8)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/2509.25944/) (2025)
- [VILTA: A VLM-in-the-Loop Adversary for Enhancing Driving Policy Robustness](https://mlsanity.com/paper/2601.12672/) (2026)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [CrashAgent: Crash Scenario Generation via Multi-modal Reasoning](https://mlsanity.com/paper/2505.18341/) (2025)
- [Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving](https://mlsanity.com/paper/2607.14387/) (2026)
- [PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving](https://mlsanity.com/paper/2609.08965/) (2026)
- [LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality](https://mlsanity.com/paper/2609.20318/) (2026)

## Abstract (arXiv)

Ensuring the safety of autonomous vehicles requires virtual scenario-based testing, which depends on the robust evaluation and generation of safety-critical scenarios. So far, researchers have used scenario-based testing frameworks that rely heavily on handcrafted scenarios as safety metrics. To reduce the effort of human interpretation and overcome the limited scalability of these approaches, we combine Large Language Models (LLMs) with structured scenario parsing and prompt engineering to automatically evaluate and generate safety-critical driving scenarios. We introduce Cartesian and Ego-centric prompt strategies for scenario evaluation, and an adversarial generation module that modifies trajectories of risk-inducing vehicles (ego-attackers) to create critical scenarios. We validate our approach using a 2D simulation framework and multiple pre-trained LLMs. The results show that the evaluation module effectively detects collision scenarios and infers scenario safety. Meanwhile, the new generation module identifies high-risk agents and synthesizes realistic, safety-critical scenarios. We conclude that an LLM equipped with domain-informed prompting techniques can effectively evaluate and generate safety-critical driving scenarios, reducing dependence on handcrafted metrics. We release our open-source code and scenarios at: https://github.com/TUM-AVS/From-Words-to-Collisions.