# From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving

arXiv: https://arxiv.org/abs/2505.22067 · HTML: https://mlsanity.com/paper/2505.22067/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 27 of 27)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [VAD: Vectorized Scene Representation for Efficient Autonomous Driving](https://mlsanity.com/paper/2303.12077/) (2023)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021)
- [VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning](https://mlsanity.com/paper/2402.13243/) (2024)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes](https://mlsanity.com/paper/2305.10430/) (2023)
- [Leapfrog Diffusion Model for Stochastic Trajectory Prediction](https://mlsanity.com/paper/2303.10895/) (2023)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Language-Guided Traffic Simulation via Scene-Level Diffusion](https://mlsanity.com/paper/2306.06344/) (2023)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)
- [Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation](https://mlsanity.com/paper/2208.00219/) (2022)
- [Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios](https://mlsanity.com/paper/2004.06531/) (2020)
- [Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies](https://mlsanity.com/paper/2006.06091/) (2020)
- [Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World](https://mlsanity.com/paper/2003.07739/) (2020)
- [Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles](https://mlsanity.com/paper/2103.07403/) (2021)
- [Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation](https://mlsanity.com/paper/2404.08570/) (2024)
- [Text2Scenario: Text-Driven Scenario Generation for Autonomous Driving Test](https://mlsanity.com/paper/2503.02911/) (2025)
- [SAFR-AV: Safety Analysis of Autonomous Vehicles using Real World Data -- An end-to-end solution for real world data driven scenario-based testing for pre-certification of AV stacks](https://mlsanity.com/paper/2302.14601/) (2023)
- [Few-Shot Testing of Autonomous Vehicles with Scenario Similarity Learning](https://mlsanity.com/paper/2409.14369/) (2024)

## Built on by (most cited first, 2 of 2)

- [DriveVLA-M0: Failure-Aware Memory Augmentation for Autonomous Driving](https://mlsanity.com/paper/2608.10413/) (2026)
- [A Local Agentic AI Framework to Generate Edge Case Scenarios for Safety Assurance](https://mlsanity.com/paper/2610.13479/) (2026)

## Abstract (arXiv)

Ensuring robust and generalizable autonomous driving requires not only broad scenario coverage but also efficient repair of failure cases, particularly those related to challenging and safety-critical scenarios. However, existing scenario generation and selection methods often lack adaptivity and semantic relevance, limiting their impact on performance improvement. In this paper, we propose SERA, an LLM-powered framework that enables autonomous driving systems to self-evolve by repairing failure cases through targeted scenario recommendation. By analyzing performance logs, SERA identifies failure patterns and dynamically retrieves semantically aligned scenarios from a structured bank. An LLM-based reflection mechanism further refines these recommendations to maximize relevance and diversity. The selected scenarios are used for few-shot fine-tuning, enabling targeted adaptation with minimal data. Experiments on the benchmark show that SERA consistently improves key metrics across multiple autonomous driving baselines, demonstrating its effectiveness and generalizability under safety-critical conditions.