# Misbehaviour Prediction for Autonomous Driving Systems

arXiv: https://arxiv.org/abs/1910.04443 · HTML: https://mlsanity.com/paper/1910.04443/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2019.

## Built on by (most cited first, 30 of 55)

- [Mind the Gap! A Study on the Transferability of Virtual vs Physical-world Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2112.11255/) (2021)
- [Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)](https://mlsanity.com/paper/2205.00664/) (2022)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [Self-Supervised Traversability Prediction by Learning to Reconstruct Safe Terrain](https://mlsanity.com/paper/2208.01329/) (2022)
- [Black-Box Testing of Deep Neural Networks Through Test Case Diversity](https://mlsanity.com/paper/2112.12591/) (2021)
- [When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study](https://mlsanity.com/paper/2212.11368/) (2022)
- [DeepGuard: A Framework for Safeguarding Autonomous Driving Systems from Inconsistent Behavior](https://mlsanity.com/paper/2111.09533/) (2021)
- [ACLS: Adaptive and Conditional Label Smoothing for Network Calibration](https://mlsanity.com/paper/2308.11911/) (2023)
- [PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing](https://mlsanity.com/paper/2111.10481/) (2021)
- [Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring](https://mlsanity.com/paper/2102.00902/) (2021)
- [Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification](https://mlsanity.com/paper/2404.18573/) (2024)
- [Unifying Evaluation of Machine Learning Safety Monitors](https://mlsanity.com/paper/2208.14660/) (2022)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Efficient Domain Augmentation for Autonomous Driving Testing Using Diffusion Models](https://mlsanity.com/paper/2409.13661/) (2024)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)
- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing](https://mlsanity.com/paper/2207.10495/) (2022)
- [How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomy](https://mlsanity.com/paper/2308.12252/) (2023)
- [Bias Behind the Wheel: Fairness Testing of Autonomous Driving Systems](https://mlsanity.com/paper/2308.02935/) (2023)
- [Boundary State Generation for Testing and Improvement of Autonomous Driving Systems](https://mlsanity.com/paper/2307.10590/) (2023)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Evaluation of Runtime Monitoring for UAV Emergency Landing](https://mlsanity.com/paper/2202.03059/) (2022)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving](https://mlsanity.com/paper/2405.01691/) (2024)
- [System Safety Monitoring of Learned Components Using Temporal Metric Forecasting](https://mlsanity.com/paper/2405.13254/) (2024)
- [Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems](https://mlsanity.com/paper/2408.13950/) (2024)
- [Benchmarking Generative AI Models for Deep Learning Test Input Generation](https://mlsanity.com/paper/2412.17652/) (2024)
- [Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines](https://mlsanity.com/paper/2212.07118/) (2022)
- [Iterative Assessment and Improvement of DNN Operational Accuracy](https://mlsanity.com/paper/2303.01295/) (2023)
- [Detecting Socially Abnormal Highway Driving Behaviors via Recurrent Graph Attention Networks](https://mlsanity.com/paper/2304.11513/) (2023)

## Abstract (arXiv)

Deep Neural Networks (DNNs) are the core component of modern autonomous driving systems. To date, it is still unrealistic that a DNN will generalize correctly in all driving conditions. Current testing techniques consist of offline solutions that identify adversarial or corner cases for improving the training phase, and little has been done for enabling online healing of DNN-based vehicles. In this paper, we address the problem of estimating the confidence of DNNs in response to unexpected execution contexts with the purpose of predicting potential safety-critical misbehaviours such as out of bound episodes or collisions. Our approach SelfOracle is based on a novel concept of self-assessment oracle, which monitors the DNN confidence at runtime, to predict unsupported driving scenarios in advance. SelfOracle uses autoencoder and time-series-based anomaly detection to reconstruct the driving scenarios seen by the car, and determine the confidence boundary of normal/unsupported conditions. In our empirical assessment, we evaluated the effectiveness of different variants of SelfOracle at predicting injected anomalous driving contexts, using DNN models and simulation environment from Udacity. Results show that, overall, SelfOracle can predict 77% misbehaviours, up to 6 seconds in advance, outperforming the online input validation approach of DeepRoad by a factor almost equal to 3.