# Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification

arXiv: https://arxiv.org/abs/2404.18573 · HTML: https://mlsanity.com/paper/2404.18573/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 73 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2204.07974/) (2022)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1811.06817/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [Comparing Offline and Online Testing of Deep Neural Networks: An Autonomous Car Case Study](https://mlsanity.com/paper/1912.00805/) (2019)
- [Towards Structured Evaluation of Deep Neural Network Supervisors](https://mlsanity.com/paper/1903.01263/) (2019)
- [Self-Checking Deep Neural Networks in Deployment](https://mlsanity.com/paper/2103.02371/) (2021)
- [DeepGuard: A Framework for Safeguarding Autonomous Driving Systems from Inconsistent Behavior](https://mlsanity.com/paper/2111.09533/) (2021)
- [Deep Predictive Models for Collision Risk Assessment in Autonomous Driving](https://mlsanity.com/paper/1711.10453/) (2017)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring](https://mlsanity.com/paper/2102.00902/) (2021)
- [Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification](https://mlsanity.com/paper/2101.00982/) (2020)

## Built on by (most cited first, 11 of 11)

- [System Safety Monitoring of Learned Components Using Temporal Metric Forecasting](https://mlsanity.com/paper/2405.13254/) (2024)
- [HybridFlow: Quantification of Aleatoric and Epistemic Uncertainty with a Single Hybrid Model](https://mlsanity.com/paper/2510.05054/) (2025)
- [Hybrid Video Anomaly Detection for Anomalous Scenarios in Autonomous Driving](https://mlsanity.com/paper/2406.06423/) (2024)
- [Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing](https://mlsanity.com/paper/2503.08936/) (2025)
- [REACT: Runtime-Enabled Active Collision-avoidance Technique for Autonomous Driving](https://mlsanity.com/paper/2505.11474/) (2025)
- [A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems](https://mlsanity.com/paper/2509.22379/) (2025)
- [Argus: Resilience-Oriented Safety Assurance Framework for End-to-End ADSs](https://mlsanity.com/paper/2511.09032/) (2025)
- [On the Assessment of Sensitivity of Autonomous Vehicle Perception](https://mlsanity.com/paper/2602.00314/) (2026)
- [GAN-enhanced Simulation-driven DNN Testing in Absence of Ground Truth](https://mlsanity.com/paper/2503.15953/) (2025)
- [Measuring What Matters: Scenario-Driven Evaluation for Trajectory Predictors in Autonomous Driving](https://mlsanity.com/paper/2512.12211/) (2025)
- [Latent Regularization in Generative Test Input Generation](https://mlsanity.com/paper/2602.15552/) (2026)

## Abstract (arXiv)

The automated real-time recognition of unexpected situations plays a crucial role in the safety of autonomous vehicles, especially in unsupported and unpredictable scenarios. This paper evaluates different Bayesian uncertainty quantification methods from the deep learning domain for the anticipatory testing of safety-critical misbehaviours during system-level simulation-based testing. Specifically, we compute uncertainty scores as the vehicle executes, following the intuition that high uncertainty scores are indicative of unsupported runtime conditions that can be used to distinguish safe from failure-inducing driving behaviors. In our study, we conducted an evaluation of the effectiveness and computational overhead associated with two Bayesian uncertainty quantification methods, namely MC- Dropout and Deep Ensembles, for misbehaviour avoidance. Overall, for three benchmarks from the Udacity simulator comprising both out-of-distribution and unsafe conditions introduced via mutation testing, both methods successfully detected a high number of out-of-bounds episodes providing early warnings several seconds in advance, outperforming two state-of-the-art misbehaviour prediction methods based on autoencoders and attention maps in terms of effectiveness and efficiency. Notably, Deep Ensembles detected most misbehaviours without any false alarms and did so even when employing a relatively small number of models, making them computationally feasible for real-time detection. Our findings suggest that incorporating uncertainty quantification methods is a viable approach for building fail-safe mechanisms in deep neural network-based autonomous vehicles.