# Fail2Drive: Benchmarking Closed-Loop Driving Generalization arXiv: https://arxiv.org/abs/2604.08535 · HTML: https://mlsanity.com/paper/2604.08535/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving. - Builds on 34 of its 39 references that are in the map. - First posted on arXiv 9 Apr 2026. ## Placed in - End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/) ## Builds on (most cited first, 30 of 34) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022) - [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023) - [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021) - [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022) - [NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking](https://mlsanity.com/paper/2406.15349/) (2024) - [Exploring the Limitations of Behavior Cloning for Autonomous Driving](https://mlsanity.com/paper/1904.08980/) (2019) - [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025) - [Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving](https://mlsanity.com/paper/2406.03877/) (2024) - [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](https://mlsanity.com/paper/2206.08129/) (2022) - [ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation](https://mlsanity.com/paper/2503.19755/) (2025) - [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021) - [SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment](https://mlsanity.com/paper/2503.09594/) (2025) - [Learning from All Vehicles](https://mlsanity.com/paper/2203.11934/) (2022) - [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025) - [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021) - [Hidden Biases of End-to-End Driving Models](https://mlsanity.com/paper/2306.07957/) (2023) - [ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles](https://mlsanity.com/paper/2405.14062/) (2024) - [HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving](https://mlsanity.com/paper/2412.01718/) (2024) - [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025) - [SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles](https://mlsanity.com/paper/2206.09682/) (2022) - [On Offline Evaluation of Vision-based Driving Models](https://mlsanity.com/paper/1809.04843/) (2018) - [NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving](https://mlsanity.com/paper/2404.07762/) (2024) - [Evaluating the Robustness of Semantic Segmentation for Autonomous Driving against Real-World Adversarial Patch Attacks](https://mlsanity.com/paper/2108.06179/) (2021) - [DTA: Physical Camouflage Attacks using Differentiable Transformation Network](https://mlsanity.com/paper/2203.09831/) (2022) - [Can Vehicle Motion Planning Generalize to Realistic Long-tail Scenarios?](https://mlsanity.com/paper/2404.07569/) (2024) - [HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder](https://mlsanity.com/paper/2503.08612/) (2025) - [CaRL: Learning Scalable Planning Policies with Simple Rewards](https://mlsanity.com/paper/2504.17838/) (2025) - [Hidden Biases of End-to-End Driving Datasets](https://mlsanity.com/paper/2412.09602/) (2024) ## Built on by (most cited first, 17 of 17) - [nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving](https://mlsanity.com/paper/2605.31572/) (2026) - [MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems](https://mlsanity.com/paper/2605.10904/) (2026) - [Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations](https://mlsanity.com/paper/2605.18059/) (2026) - [BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2606.08684/) (2026) - [Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos](https://mlsanity.com/paper/2606.07366/) (2026) - [Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City](https://mlsanity.com/paper/2606.20980/) (2026) - [Herding End-to-End Autonomous Driving via Neuro-Symbolic Safety Guards](https://mlsanity.com/paper/2608.11451/) (2026) - [Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms](https://mlsanity.com/paper/2608.20111/) (2026) - [RoG-DAgger: Rollout-Guided Post-Training for End-to-End Driving](https://mlsanity.com/paper/2608.24525/) (2026) - [LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings](https://mlsanity.com/paper/2609.06368/) (2026) - [FIVE-VLA: Fast and EffectIVE Autonomous Driving with Recurrent Action Memory](https://mlsanity.com/paper/2609.18623/) (2026) - [NavSafe-∞: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments](https://mlsanity.com/paper/2609.26618/) (2026) - [DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving](https://mlsanity.com/paper/2609.26792/) (2026) - [Progressive Risk Estimation for Accident Anticipation](https://mlsanity.com/paper/2609.32811/) (2026) - [VehDyn: A Driving World Model Benchmark for Vehicle Dynamics](https://mlsanity.com/paper/2609.33264/) (2026) - [Localized Conformal Safety Monitoring with Vision-Language Models for Autonomous Driving](https://mlsanity.com/paper/2610.02765/) (2026) - [VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoning](https://mlsanity.com/paper/2610.08761/) (2026) ## Abstract (arXiv) Generalization under distribution shift remains a central bottleneck for closed-loop autonomous driving. Although simulators like CARLA enable safe and scalable testing, existing benchmarks rarely measure true generalization: they typically reuse training scenarios at test time. Success can therefore reflect memorization rather than robust driving behavior. We introduce Fail2Drive, the first paired-route benchmark for closed-loop generalization in CARLA, with 200 routes and 17 new scenario classes spanning appearance, layout, behavioral, and robustness shifts. Each shifted route is matched with an in-distribution counterpart, isolating the effect of the shift and turning qualitative failures into quantitative diagnostics. Evaluating multiple state-of-the-art models reveals consistent degradation, with an average success-rate drop of 22.8%. Our analysis uncovers unexpected failure modes, such as ignoring objects clearly visible in the LiDAR and failing to learn the fundamental concepts of free and occupied space. To accelerate follow-up work, Fail2Drive includes an open-source toolbox for creating new scenarios and validating solvability via a privileged expert policy. Together, these components establish a reproducible foundation for benchmarking and improving closed-loop driving generalization. We open-source all code, data, and tools at https://github.com/autonomousvision/fail2drive .