# Provably-Robust Runtime Monitoring of Neuron Activation Patterns

arXiv: https://arxiv.org/abs/2011.11959 · HTML: https://mlsanity.com/paper/2011.11959/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 9 of 9)

- [Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networks](https://mlsanity.com/paper/2212.07773/) (2022)
- [The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities](https://mlsanity.com/paper/2510.10238/) (2025)
- [Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes](https://mlsanity.com/paper/2104.14435/) (2021)
- [SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networks](https://mlsanity.com/paper/2106.01917/) (2021)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [Edge Case Detection in Automated Driving: Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2410.08491/) (2024)
- [Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features](https://mlsanity.com/paper/2103.03704/) (2021)
- [Runtime Monitoring DNN-Based Perception](https://mlsanity.com/paper/2310.03999/) (2023)
- [Self-Aware Object Detection via Degradation Manifolds](https://mlsanity.com/paper/2602.18394/) (2026)

## Abstract (arXiv)

For deep neural networks (DNNs) to be used in safety-critical autonomous driving tasks, it is desirable to monitor in operation time if the input for the DNN is similar to the data used in DNN training. While recent results in monitoring DNN activation patterns provide a sound guarantee due to building an abstraction out of the training data set, reducing false positives due to slight input perturbation has been an issue towards successfully adapting the techniques. We address this challenge by integrating formal symbolic reasoning inside the monitor construction process. The algorithm performs a sound worst-case estimate of neuron values with inputs (or features) subject to perturbation, before the abstraction function is applied to build the monitor. The provable robustness is further generalized to cases where monitoring a single neuron can use more than one bit, implying that one can record activation patterns with a fine-grained decision on the neuron value interval.