mlsanityData to 7 Oct 2026

NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Driving

arXiv 2404.07762 ↗First posted 11 Apr 2024cs.CVcited by 61

On the map: End-to-end autonomous driving → Adversarial scenarios and safety testing

Abstract (arXiv)

We present a versatile NeRF-based simulator for testing autonomous driving (AD) software systems, designed with a focus on sensor-realistic closed-loop evaluation and the creation of safety-critical scenarios. The simulator learns from sequences of real-world driving sensor data and enables reconfigurations and renderings of new, unseen scenarios. In this work, we use our simulator to test the responses of AD models to safety-critical scenarios inspired by the European New Car Assessment Programme (Euro NCAP). Our evaluation reveals that, while state-of-the-art end-to-end planners excel in nominal driving scenarios in an open-loop setting, they exhibit critical flaws when navigating our safety-critical scenarios in a closed-loop setting. This highlights the need for advancements in the safety and real-world usability of end-to-end planners. By publicly releasing our simulator and scenarios as an easy-to-run evaluation suite, we invite the research community to explore, refine, and validate their AD models in controlled, yet highly configurable and challenging sensor-realistic environments. Code and instructions can be found at https://github.com/atonderski/neuro-ncap

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving. Builds on 29 of its 41 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

29 of 41 references.

References · 29

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  7. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  8. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  9. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  10. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  11. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  12. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
17 more
  1. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  2. 2021Multi-Modal Fusion Transformer for End-to-End Autonomous Drivingcited by 372
  3. 2022ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learningcited by 365
  4. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  5. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  6. 2023UniSim: A Neural Closed-Loop Sensor Simulatorcited by 294
  7. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  8. 2023Parting with Misconceptions about Learning-based Vehicle Motion Planningcited by 238
  9. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  10. 2021PandaSet: Advanced Sensor Suite Dataset for Autonomous Drivingcited by 186
  11. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  12. 2023NeuRAD: Neural Rendering for Autonomous Drivingcited by 146
  13. 2021Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversalscited by 112
  14. 2024Towards learning-based planning:The nuPlan benchmark for real-world autonomous drivingcited by 103
  15. 2023Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous drivingcited by 83
  16. 2018On Offline Evaluation of Vision-based Driving Modelscited by 62
  17. 2022Waymo Open Dataset: Panoramic Video Panoptic Segmentationcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 31; 2026: 23.

61 papers.

Sort:

By area:

  1. 2025Pseudo-Simulation for Autonomous Drivingcited by 141
  2. 2024HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Drivingcited by 84
  3. 2024SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Drivingcited by 66
  4. 2025Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Modelscited by 60
  5. 2025RAP: 3D Rasterization Augmented End-to-End Planningcited by 39
  6. 2024DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulationcited by 34
  7. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  8. 2025SimScale: Learning to Drive via Real-World Simulation at Scalecited by 34
  9. 2024G3R: Gradient Guided Generalizable Reconstructioncited by 29
  10. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  11. 2025CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Drivingcited by 25
  12. 2025ImagiDrive: A Unified Imagination-and-Planning Framework for Autonomous Drivingcited by 24
38 more
  1. 2025LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Drivingcited by 22
  2. 2025MTGS: Multi-Traversal Gaussian Splattingcited by 17
  3. 2025R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulationcited by 17
  4. 2025Challenger: Affordable Adversarial Driving Video Generationcited by 16
  5. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  6. 2024SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIscited by 14
  7. 2025Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Drivingcited by 14
  8. 2025Unraveling the Effects of Synthetic Data on End-to-End Autonomous Drivingcited by 12
  9. 2025Decoupled Diffusion Sparks Adaptive Scene Generationcited by 12
  10. 2025RealEngine: Simulating Autonomous Driving in Realistic Contextcited by 10
  11. 2025ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Drivingcited by 10
  12. 2025Learning-based 3D Reconstruction in Autonomous Driving: A Comprehensive Surveycited by 9
  13. 2024UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulationcited by 7
  14. 2025NeuRadar: Neural Radiance Fields for Automotive Radar Point Cloudscited by 7
  15. 2025Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Drivingcited by 7
  16. 2025Divide and Merge: Motion and Semantic Learning in End-to-End Autonomous Drivingcited by 5
  17. 2025B'ezierGS: Dynamic Urban Scene Reconstruction with B'ezier Curve Gaussian Splattingcited by 5
  18. 2025CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Modelcited by 5
  19. 2024S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Drivingcited by 4
  20. 2026Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Datasetcited by 4
  21. 2026Scaling Self-Play for End-to-End Drivingcited by 4
  22. 2025Industrial-Grade Sensor Simulation via Gaussian Splatting: A Modular Framework for Scalable Editing and Full-Stack Validationcited by 3
  23. 2025Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulationcited by 3
  24. 2025IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenescited by 3
  25. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 3
  26. 2026A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous drivingcited by 3
  27. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  28. 2026Pictura: Perspective-View Self-Play at Scale for Drivingcited by 2
  29. 2025Collision Risk Estimation via Loss Prediction in End-to-End Autonomous Drivingcited by 1
  30. 2026Unified Sensor Simulation for Autonomous Drivingcited by 1
  31. 2026Class-Incremental Motion Forecastingcited by 1
  32. 2026ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagationcited by 1
  33. 2026ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Drivingcited by 1
  34. 2025RoCo-Sim: Enhancing Roadside Collaborative Perception through Foreground Simulationcited by 0
  35. 2025ToosiCubix: Monocular 3D Cuboid Labeling via Vehicle Part Annotationscited by 0
  36. 2025Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environmentscited by 0
  37. 2026Enhancing Novel View Synthesis via Geometry Grounded Set Diffusioncited by 0
  38. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representationscited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Driving
  2. 2025A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems
  3. 2025Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarios
  4. 2026NavSafe-∞: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments
  5. 2024Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap
  6. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles
  7. 2025Bench2ADVLM: A Closed-Loop Benchmark for Vision-language Models in Autonomous Driving
  8. 2026From Virtual Environments to Real-World Trials: Emerging Trends in Autonomous Driving