# Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey

arXiv: https://arxiv.org/abs/2511.05982 · HTML: https://mlsanity.com/paper/2511.05982/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 8 Nov 2025.

## Builds on (most cited first, 30 of 33)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [Generalization Error in Deep Learning](https://mlsanity.com/paper/1808.01174/) (2018)
- [Outside the Box: Abstraction-Based Monitoring of Neural Networks](https://mlsanity.com/paper/1911.09032/) (2019)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Runtime Monitoring Neuron Activation Patterns](https://mlsanity.com/paper/1809.06573/) (2018)
- [Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Tasks](https://mlsanity.com/paper/2001.08001/) (2020)
- [Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends](https://mlsanity.com/paper/2101.01364/) (2021)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness](https://mlsanity.com/paper/1905.11427/) (2019)
- [Into the Unknown: Active Monitoring of Neural Networks](https://mlsanity.com/paper/2009.06429/) (2020)
- [DeepGuard: A Framework for Safeguarding Autonomous Driving Systems from Inconsistent Behavior](https://mlsanity.com/paper/2111.09533/) (2021)
- [Detecting Adversarial Perturbations in Multi-Task Perception](https://mlsanity.com/paper/2203.01177/) (2022)
- [Adversarial Attacks on Traffic Sign Recognition: A Survey](https://mlsanity.com/paper/2307.08278/) (2023)
- [Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networks](https://mlsanity.com/paper/2212.07773/) (2022)
- [Real-Time Detectors for Digital and Physical Adversarial Inputs to Perception Systems](https://mlsanity.com/paper/2002.09792/) (2020)
- [Suppress with a Patch: Revisiting Universal Adversarial Patch Attacks against Object Detection](https://mlsanity.com/paper/2209.13353/) (2022)
- [A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networks](https://mlsanity.com/paper/2310.20349/) (2023)
- [BAM: Box Abstraction Monitors for Real-time OoD Detection in Object Detection](https://mlsanity.com/paper/2403.18373/) (2024)
- [Increasing Trustworthiness of Deep Neural Networks via Accuracy Monitoring](https://mlsanity.com/paper/2007.01472/) (2020)
- [Adversarial Vulnerability of Temporal Feature Networks for Object Detection](https://mlsanity.com/paper/2208.10773/) (2022)

## Built on by (most cited first, 2 of 2)

- [Lost in Fog: Sensor Perturbations Expose Reasoning Fragility in Driving VLAs](https://mlsanity.com/paper/2605.21446/) (2026)
- [Knowledge-Guided Failure Prediction: Detecting When Object Detectors Miss Safety-Critical Objects](https://mlsanity.com/paper/2603.25499/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) are widely used in perception systems for safety-critical applications, such as autonomous driving and robotics. However, DNNs remain vulnerable to various safety concerns, including generalization errors, out-of-distribution (OOD) inputs, and adversarial attacks, which can lead to hazardous failures. This survey provides a comprehensive overview of runtime safety monitoring approaches, which operate in parallel to DNNs during inference to detect these safety concerns without modifying the DNN itself. We categorize existing methods into three main groups: Monitoring inputs, internal representations, and outputs. We analyze the state-of-the-art for each category, identify strengths and limitations, and map methods to the safety concerns they address. In addition, we highlight open challenges and future research directions.