# Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

arXiv: https://arxiv.org/abs/2003.07739 · HTML: https://mlsanity.com/paper/2003.07739/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Mar 2020.

## Built on by (most cited first, 30 of 39)

- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [RaLiBEV: Radar and LiDAR BEV Fusion Learning for Anchor Box Free Object Detection Systems](https://mlsanity.com/paper/2211.06108/) (2022)
- [Statistical Perspectives on Reliability of Artificial Intelligence Systems](https://mlsanity.com/paper/2111.05391/) (2021)
- [Editing Driver Character: Socially-Controllable Behavior Generation for Interactive Traffic Simulation](https://mlsanity.com/paper/2303.13830/) (2023)
- [LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends](https://mlsanity.com/paper/2410.15281/) (2024)
- [Parallel and Multi-Objective Falsification with Scenic and VerifAI](https://mlsanity.com/paper/2107.04164/) (2021)
- [A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation](https://mlsanity.com/paper/2110.14870/) (2021)
- [Formalizing and Evaluating Requirements of Perception Systems for Automated Vehicles using Spatio-Temporal Perception Logic](https://mlsanity.com/paper/2206.14372/) (2022)
- [Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis](https://mlsanity.com/paper/2211.02736/) (2022)
- [Coverage-based Scene Fuzzing for Virtual Autonomous Driving Testing](https://mlsanity.com/paper/2106.00873/) (2021)
- [Continual Driving Policy Optimization with Closed-Loop Individualized Curricula](https://mlsanity.com/paper/2309.14209/) (2023)
- [Querying Labelled Data with Scenario Programs for Sim-to-Real Validation](https://mlsanity.com/paper/2112.00206/) (2021)
- [SAFR-AV: Safety Analysis of Autonomous Vehicles using Real World Data -- An end-to-end solution for real world data driven scenario-based testing for pre-certification of AV stacks](https://mlsanity.com/paper/2302.14601/) (2023)
- [Scenario-based Compositional Verification of Autonomous Systems with Neural Perception](https://mlsanity.com/paper/2504.20942/) (2025)
- [Pedestrian and Passenger Interaction with Autonomous Vehicles: Field Study in a Crosswalk Scenario](https://mlsanity.com/paper/2312.07606/) (2023)
- [From Failures to Fixes: LLM-Driven Scenario Repair for Self-Evolving Autonomous Driving](https://mlsanity.com/paper/2505.22067/) (2025)
- [SilGAN: Generating driving maneuvers for scenario-based software-in-the-loop testing](https://mlsanity.com/paper/2107.07364/) (2021)
- [A Training Rate and Survival Heuristic for Inference and Robustness Evaluation (TRASHFIRE)](https://mlsanity.com/paper/2401.13751/) (2024)
- [Representation Alignment Contrastive Regularization for Multi-Object Tracking](https://mlsanity.com/paper/2404.02562/) (2024)
- [Flow to Rare Events: An Application of Normalizing Flow in Temporal Importance Sampling for Automated Vehicle Validation](https://mlsanity.com/paper/2407.07320/) (2024)
- [AuthSim: Towards Authentic and Effective Safety-critical Scenario Generation for Autonomous Driving Tests](https://mlsanity.com/paper/2502.21100/) (2025)
- [Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety](https://mlsanity.com/paper/2503.00077/) (2025)
- [A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving](https://mlsanity.com/paper/2506.14100/) (2025)
- [Warm Starting State-Space Models with Automata Learning](https://mlsanity.com/paper/2603.05694/) (2026)
- [Resolving space-sharing conflicts in road user interactions through uncertainty reduction: An active inference-based computational model](https://mlsanity.com/paper/2604.19838/) (2026)
- [Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective](https://mlsanity.com/paper/2607.05783/) (2026)

## Abstract (arXiv)

We present a new approach to automated scenario-based testing of the safety of autonomous vehicles, especially those using advanced artificial intelligence-based components, spanning both simulation-based evaluation as well as testing in the real world. Our approach is based on formal methods, combining formal specification of scenarios and safety properties, algorithmic test case generation using formal simulation, test case selection for track testing, executing test cases on the track, and analyzing the resulting data. Experiments with a real autonomous vehicle at an industrial testing facility support our hypotheses that (i) formal simulation can be effective at identifying test cases to run on the track, and (ii) the gap between simulated and real worlds can be systematically evaluated and bridged.