# Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction

arXiv: https://arxiv.org/abs/2008.06020 · HTML: https://mlsanity.com/paper/2008.06020/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 12 of 12)

- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles](https://mlsanity.com/paper/2109.06126/) (2021)
- [DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars](https://mlsanity.com/paper/2209.12729/) (2022)
- [What data do we need for training an AV motion planner?](https://mlsanity.com/paper/2105.12337/) (2021)
- [Towards Fairer and More Efficient Federated Learning via Multidimensional Personalized Edge Models](https://mlsanity.com/paper/2302.04464/) (2023)
- [UniGen: Unified Modeling of Initial Agent States and Trajectories for Generating Autonomous Driving Scenarios](https://mlsanity.com/paper/2405.03807/) (2024)
- [A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation](https://mlsanity.com/paper/2110.14870/) (2021)
- [Validation of Composite Systems by Discrepancy Propagation](https://mlsanity.com/paper/2210.12061/) (2022)
- [Perception Imitation: Towards Synthesis-free Simulator for Autonomous Vehicles](https://mlsanity.com/paper/2304.09365/) (2023)
- [EMPERROR: A Flexible Generative Perception Error Model for Probing Self-Driving Planners](https://mlsanity.com/paper/2411.07719/) (2024)
- [Quantity over Quality: Training an AV Motion Planner with Large Scale Commodity Vision Data](https://mlsanity.com/paper/2203.01681/) (2022)

## Abstract (arXiv)

We present a novel method for testing the safety of self-driving vehicles in simulation. We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. Instead, we directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing. Specifically, we use paired data in the form of ground truth labels and real perception and prediction outputs to train a model that predicts what the online system will produce. Importantly, the inputs to our system consists of high definition maps, bounding boxes, and trajectories, which can be easily sketched by a test engineer in a matter of minutes. This makes our approach a much more scalable solution. Quantitative results on two large-scale datasets demonstrate that we can realistically test motion planning using our simulations.