# A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

arXiv: https://arxiv.org/abs/2112.00964 · HTML: https://mlsanity.com/paper/2112.00964/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 140 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 52)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving](https://mlsanity.com/paper/1505.00256/) (2015)
- [Formal Security Analysis of Neural Networks using Symbolic Intervals](https://mlsanity.com/paper/1804.10829/) (2018)
- [Efficient Formal Safety Analysis of Neural Networks](https://mlsanity.com/paper/1809.08098/) (2018)
- [Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving](https://mlsanity.com/paper/1907.06826/) (2019)
- [AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles](https://mlsanity.com/paper/2101.06549/) (2021)
- [On a Formal Model of Safe and Scalable Self-driving Cars](https://mlsanity.com/paper/1708.06374/) (2017)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation](https://mlsanity.com/paper/1811.00145/) (2018)
- [Learning to Collide: An Adaptive Safety-Critical Scenarios Generating Method](https://mlsanity.com/paper/2003.01197/) (2020)
- [A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems](https://mlsanity.com/paper/2005.02979/) (2020)
- [Dirty Road Can Attack: Security of Deep Learning based Automated Lane Centering under Physical-World Attack](https://mlsanity.com/paper/2009.06701/) (2020)
- [Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation](https://mlsanity.com/paper/2009.08311/) (2020)
- [DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems](https://mlsanity.com/paper/1812.10812/) (2018)
- [Scenarios for Development, Test and Validation of Automated Vehicles](https://mlsanity.com/paper/1801.08598/) (2018)
- [Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniques](https://mlsanity.com/paper/1605.04965/) (2016)
- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios](https://mlsanity.com/paper/2004.06531/) (2020)
- [Ontology based Scene Creation for the Development of Automated Vehicles](https://mlsanity.com/paper/1704.01006/) (2017)
- [Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving](https://mlsanity.com/paper/1903.10654/) (2019)
- [Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2008.10581/) (2020)

## Built on by (most cited first, 21 of 21)

- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [Rethinking Closed-loop Training for Autonomous Driving](https://mlsanity.com/paper/2306.15713/) (2023)
- [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024)
- [Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain](https://mlsanity.com/paper/2307.11769/) (2023)
- [Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives](https://mlsanity.com/paper/2505.09074/) (2025)
- [Continual Driving Policy Optimization with Closed-Loop Individualized Curricula](https://mlsanity.com/paper/2309.14209/) (2023)
- [ROCAS: Root Cause Analysis of Autonomous Driving Accidents via Cyber-Physical Co-mutation](https://mlsanity.com/paper/2409.07774/) (2024)
- [Rate-Informed Discovery via Bayesian Adaptive Multifidelity Sampling](https://mlsanity.com/paper/2411.17826/) (2024)
- [Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration](https://mlsanity.com/paper/2402.07031/) (2024)
- [Dance of the ADS: Orchestrating Failures through Historically-Informed Scenario Fuzzing](https://mlsanity.com/paper/2407.04359/) (2024)
- [Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety](https://mlsanity.com/paper/2503.00077/) (2025)
- [Semantic and Articulated Pedestrian Sensing Onboard a Moving Vehicle](https://mlsanity.com/paper/2309.06313/) (2023)
- [Towards provable probabilistic safety for scalable embodied AI systems](https://mlsanity.com/paper/2506.05171/) (2025)
- [Bayesian Optimization applied for accelerated Virtual Validation of the Autonomous Driving Function](https://mlsanity.com/paper/2507.22769/) (2025)
- [SCOUT: A Lightweight Framework for Scenario Coverage Assessment in Autonomous Driving](https://mlsanity.com/paper/2510.24949/) (2025)
- [Declarative Scenario-based Testing with RoadLogic](https://mlsanity.com/paper/2603.09455/) (2026)
- [Emergency Lane-Change Simulation: A Behavioral Guidance Approach for Risky Scenario Generation](https://mlsanity.com/paper/2603.20234/) (2026)
- [From Nominal Intensity to Equivalent Rainfall: A Path-Based Credibility Evaluation Framework for Simulated Rainfall in Autonomous-Driving Perception Tests](https://mlsanity.com/paper/2606.11989/) (2026)
- [Revealing Safety-Critical Scenarios for UTM via Transformer](https://mlsanity.com/paper/2606.31114/) (2026)
- [CUBICS: Situation-aware performance estimation for safety-relevant ML components](https://mlsanity.com/paper/2608.16564/) (2026)
- [Evaluating Physical Consistency and Plausibility in Generative Scenario Models for Autonomous Driving](https://mlsanity.com/paper/2610.01581/) (2026)

## Abstract (arXiv)

Automated Driving Systems (ADSs) have seen rapid progress in recent years. To ensure the safety and reliability of these systems, extensive testings are being conducted before their future mass deployment. Testing the system on the road is the closest to real-world and desirable approach, but it is incredibly costly. Also, it is infeasible to cover rare corner cases using such real-world testing. Thus, a popular alternative is to evaluate an ADS's performance in some well-designed challenging scenarios, a.k.a. scenario-based testing. High-fidelity simulators have been widely used in this setting to maximize flexibility and convenience in testing what-if scenarios. Although many works have been proposed offering diverse frameworks/methods for testing specific systems, the comparisons and connections among these works are still missing. To bridge this gap, in this work, we provide a generic formulation of scenario-based testing in high-fidelity simulation and conduct a literature review on the existing works. We further compare them and present the open challenges as well as potential future research directions.