mlsanityData to 7 Oct 2026

SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles

arXiv 2206.09682 ↗First posted 20 Jun 2022cs.ROcited by 65

On the map: End-to-end autonomous driving → Adversarial scenarios and safety testing

Abstract (arXiv)

As shown by recent studies, machine intelligence-enabled systems are vulnerable to test cases resulting from either adversarial manipulation or natural distribution shifts. This has raised great concerns about deploying machine learning algorithms for real-world applications, especially in safety-critical domains such as autonomous driving (AD). On the other hand, traditional AD testing on naturalistic scenarios requires hundreds of millions of driving miles due to the high dimensionality and rareness of the safety-critical scenarios in the real world. As a result, several approaches for autonomous driving evaluation have been explored, which are usually, however, based on different simulation platforms, types of safety-critical scenarios, scenario generation algorithms, and driving route variations. Thus, despite a large amount of effort in autonomous driving testing, it is still challenging to compare and understand the effectiveness and efficiency of different testing scenario generation algorithms and testing mechanisms under similar conditions. In this paper, we aim to provide the first unified platform SafeBench to integrate different types of safety-critical testing scenarios, scenario generation algorithms, and other variations such as driving routes and environments. Meanwhile, we implement 4 deep reinforcement learning-based AD algorithms with 4 types of input (e.g., bird's-eye view, camera) to perform fair comparisons on SafeBench. We find our generated testing scenarios are indeed more challenging and observe the trade-off between the performance of AD agents under benign and safety-critical testing scenarios. We believe our unified platform SafeBench for large-scale and effective autonomous driving testing will motivate the development of new testing scenario generation and safe AD algorithms. SafeBench is available at https://safebench.github.io.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving. Builds on 45 of its 114 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

45 of 114 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks17 Jun

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
32 more
  1. 2017Adversarial Attacks on Neural Network Policiescited by 314
  2. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  3. 2020PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentationcited by 274
  4. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  5. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  6. 2020SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentationcited by 170
  7. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  8. 2018Generating 3D Adversarial Point Cloudscited by 144
  9. 2021AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehiclescited by 132
  10. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  11. 2020Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentationcited by 112
  12. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  13. 2019Model-free Deep Reinforcement Learning for Urban Autonomous Drivingcited by 96
  14. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 89
  15. 2021VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehiclescited by 81
  16. 2018Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulationcited by 76
  17. 2020Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Methodcited by 74
  18. 2018CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognitioncited by 63
  19. 2020Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluationcited by 54
  20. 2020TSS: Transformation-Specific Smoothing for Robustness Certificationcited by 49
  21. 2016Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniquescited by 44
  22. 2017Ontology based Scene Creation for the Development of Automated Vehiclescited by 40
  23. 2019SUMMIT: A Simulator for Urban Driving in Massive Mixed Trafficcited by 39
  24. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  25. 2021Learn-to-Race: A Multimodal Control Environment for Autonomous Racingcited by 23
  26. 2021CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoningcited by 23
  27. 2019CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarioscited by 19
  28. 2020Deep Probabilistic Accelerated Evaluation: A Robust Certifiable Rare-Event Simulation Methodology for Black-Box Safety-Critical Systemscited by 6
  29. 2017A Versatile Approach to Evaluating and Testing Automated Vehicles based on Kernel Methodscited by 5
  30. 2021AutoDRIVE Simulator: A Simulator for Scaled Autonomous Vehicle Research and Educationcited by 5
  31. 2017Sequential Experimentation to Efficiently Test Automated Vehiclescited by 4
  32. 2018Developing Robot Driver Etiquette Based on Naturalistic Human Driving Behaviorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 9; 2025: 22; 2026: 22.

65 papers.

Sort:

By area:

  1. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  3. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  4. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 92
  5. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  7. 2023CAT: Closed-loop Adversarial Training for Safe End-to-End Drivingcited by 52
  8. 2023Synthetic Datasets for Autonomous Driving: A Surveycited by 48
  9. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  10. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  11. 2026Fail2Drive: Benchmarking Closed-Loop Driving Generalizationcited by 15
  12. 2024Traffic Scene Generation from Natural Language Description for Autonomous Vehicles with Large Language Modelcited by 14
38 more
  1. 2024FREA: Feasibility-Guided Generation of Safety-Critical Scenarios with Reasonable Adversarialitycited by 13
  2. 2023Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenescited by 12
  3. 2023RiskBench: A Scenario-based Benchmark for Risk Identificationcited by 12
  4. 2024CaDRE: Controllable and Diverse Generation of Safety-Critical Driving Scenarios using Real-World Trajectoriescited by 11
  5. 2024COLEP: Certifiably Robust Learning-Reasoning Conformal Prediction via Probabilistic Circuitscited by 9
  6. 2023(Re)2H2O: Autonomous Driving Scenario Generation via Reversely Regularized Hybrid Offline-and-Online Reinforcement Learningcited by 6
  7. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  8. 2025DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulationcited by 5
  9. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  10. 2023Exploring the Potential of Multi-Modal AI for Driving Hazard Predictioncited by 4
  11. 2025RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulationcited by 4
  12. 2025Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Modelscited by 4
  13. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehiclescited by 4
  14. 2025AirTrafficGen: Configurable Air Traffic Scenario Generation with Large Language Modelscited by 3
  15. 2025MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Drivingcited by 3
  16. 2025Controllable Collision Scenario Generation via Collision Pattern Predictioncited by 3
  17. 2026CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluationcited by 3
  18. 2026ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Drivingcited by 3
  19. 2023Pixel-wise Smoothing for Certified Robustness against Camera Motion Perturbationscited by 2
  20. 2024Dance of the ADS: Orchestrating Failures through Historically-Informed Scenario Fuzzingcited by 2
  21. 2024Data-driven Diffusion Models for Enhancing Safety in Autonomous Vehicle Traffic Simulationscited by 2
  22. 2025ATARS: An Aerial Traffic Atomic Activity Recognition and Temporal Segmentation Datasetcited by 2
  23. 2025AccidentSim: Generating Vehicle Collision Videos with Physically Realistic Collision Trajectories from Real-World Accident Reportscited by 2
  24. 2025Safety2Drive: Safety-Critical Scenario Benchmark for the Evaluation of Autonomous Drivingcited by 2
  25. 2025From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Modelscited by 2
  26. 2025OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Modelscited by 2
  27. 2026ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Modelscited by 2
  28. 2024VCAT: Vulnerability-aware and Curiosity-driven Adversarial Training for Enhancing Autonomous Vehicle Robustnesscited by 1
  29. 2025Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safetycited by 1
  30. 2025Towards Benchmarking and Assessing the Safety and Robustness of Autonomous Driving on Safety-critical Scenarioscited by 1
  31. 2025My Precious Crash Data: Barriers and Opportunities in Encouraging Autonomous Driving Companies to Share Safety-Critical Datacited by 1
  32. 2025ADReFT: Adaptive Decision Repair for Safe Autonomous Driving via Reinforcement Fine-Tuningcited by 1
  33. 2025Reimagining Safety Alignment with An Imagecited by 1
  34. 2022Scoring Black-Box Models for Adversarial Robustnesscited by 0
  35. 2025EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Drivingcited by 0
  36. 2025CORTEX-AVD: A Framework for CORner Case Testing and EXploration in Autonomous Vehicle Developmentcited by 0
  37. 2026Uncertainty-Aware Vision-based Risk Object Identification via Conformal Risk Tube Predictioncited by 0
  38. 2026ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Drivingcited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SaFeR: Safety-Critical Scenario Generation for Autonomous Driving Test via Feasibility-Constrained Token Resampling
  2. 2024Generating Critical Scenarios for Testing Automated Driving Systems
  3. 2025Enhancing Autonomous Driving Safety with Collision Scenario Integration
  4. 2025AuthSim: Towards Authentic and Effective Safety-critical Scenario Generation for Autonomous Driving Tests
  5. 2025From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios
  6. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Driving
  7. 2025SAFE: Harnessing LLM for Scenario-Driven ADS Testing from Multimodal Crash Data ↗
  8. 2025DSBench: A Comprehensive Benchmark for Evaluating External and In-Cabin Risks