# LLM-attacker: Enhancing Closed-loop Adversarial Scenario Generation for Autonomous Driving with Large Language Models

arXiv: https://arxiv.org/abs/2501.15850 · HTML: https://mlsanity.com/paper/2501.15850/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 23 of 23)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior](https://mlsanity.com/paper/2112.05077/) (2021)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [LLM4Drive: A Survey of Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.01043/) (2023)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation](https://mlsanity.com/paper/2404.08570/) (2024)
- [Geolocation Representation from Large Language Models are Generic Enhancers for Spatio-Temporal Learning](https://mlsanity.com/paper/2408.12116/) (2024)
- [Accelerated Evaluation of Automated Vehicles Using Piecewise Mixture Models](https://mlsanity.com/paper/1701.08915/) (2017)
- [Adaptive Safety Evaluation for Connected and Automated Vehicles with Sparse Control Variates](https://mlsanity.com/paper/2212.00517/) (2022)
- [Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving](https://mlsanity.com/paper/2409.07321/) (2024)
- [Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies](https://mlsanity.com/paper/2410.10212/) (2024)
- [High-Dimensional Fault Tolerance Testing of Highly Automated Vehicles Based on Low-Rank Models](https://mlsanity.com/paper/2407.21069/) (2024)

## Built on by (most cited first, 19 of 19)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmap](https://mlsanity.com/paper/2503.21411/) (2025)
- [HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defense](https://mlsanity.com/paper/2601.04034/) (2026)
- [Seeking to Collide: Online Safety-Critical Scenario Generation for Autonomous Driving with Retrieval Augmented Large Language Models](https://mlsanity.com/paper/2505.00972/) (2025)
- [CrashAgent: Crash Scenario Generation via Multi-modal Reasoning](https://mlsanity.com/paper/2505.18341/) (2025)
- [Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models](https://mlsanity.com/paper/2512.20677/) (2025)
- [VLM as Strategist: Adaptive Generation of Safety-critical Testing Scenarios via Guided Diffusion](https://mlsanity.com/paper/2512.02844/) (2025)
- [Virtual Traffic Police: Large Language Model-Augmented Traffic Signal Control for Unforeseen Incidents](https://mlsanity.com/paper/2601.15816/) (2026)
- [Risk-Controllable Multi-View Diffusion for Driving Scenario Generation](https://mlsanity.com/paper/2603.11534/) (2026)
- [LLM-Generated Fault Scenarios for Evaluating Perception-Driven Lane Following in Autonomous Edge Systems](https://mlsanity.com/paper/2604.07362/) (2026)
- [World Models as Adversaries: Multi-Agent Self-Play Fine-Tuning for Robust Motion Planning](https://mlsanity.com/paper/2607.10630/) (2026)
- [Robustness of LLM-enabled vehicle trajectory prediction under data security threats](https://mlsanity.com/paper/2511.13753/) (2025)
- [KG-ASG: Collision-Knowledge-Guided Closed-Loop Adversarial Scenario Generation With Primary-Support Attribution](https://mlsanity.com/paper/2605.18895/) (2026)
- [EvoDrive: Pareto Evolution for Safety-Critical Autonomous Driving via Self-Improving LLM Agents](https://mlsanity.com/paper/2606.03678/) (2026)
- [SPHINX: First Explain, Then Explore](https://mlsanity.com/paper/2606.17482/) (2026)
- [TACTIC: Temporal and Context-Aware LLM Tactical Planning for Roadside LiDAR Attacks](https://mlsanity.com/paper/2609.39969/) (2026)
- [Human Behavior-Informed Crash Scenario Generation with Real-World Crash Priors for Autonomous Vehicle Safety Evaluation](https://mlsanity.com/paper/2610.04366/) (2026)
- [Lamarck's Driving School: Discovering Autonomous Driving Training Strategies through Evolutionary Competition](https://mlsanity.com/paper/2610.11662/) (2026)
- [A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance](https://mlsanity.com/paper/2610.13479/) (2026)

## Abstract (arXiv)

Ensuring and improving the safety of autonomous driving systems (ADS) is crucial for the deployment of highly automated vehicles, especially in safety-critical events. To address the rarity issue, adversarial scenario generation methods are developed, in which behaviors of traffic participants are manipulated to induce safety-critical events. However, existing methods still face two limitations. First, identification of the adversarial participant directly impacts the effectiveness of the generation. However, the complexity of real-world scenarios, with numerous participants and diverse behaviors, makes identification challenging. Second, the potential of generated safety-critical scenarios to continuously improve ADS performance remains underexplored. To address these issues, we propose LLM-attacker: a closed-loop adversarial scenario generation framework leveraging large language models (LLMs). Specifically, multiple LLM agents are designed and coordinated to identify optimal attackers. Then, the trajectories of the attackers are optimized to generate adversarial scenarios. These scenarios are iteratively refined based on the performance of ADS, forming a feedback loop to improve ADS. Experimental results show that LLM-attacker can create more dangerous scenarios than other methods, and the ADS trained with it achieves a collision rate half that of training with normal scenarios. This indicates the ability of LLM-attacker to test and enhance the safety and robustness of ADS. Video demonstrations are provided at: https://drive.google.com/file/d/1Zv4V3iG7825oyiKbUwS2Y-rR0DQIE1ZA/view.