# Outside the Box: Abstraction-Based Monitoring of Neural Networks

arXiv: https://arxiv.org/abs/1911.09032 · HTML: https://mlsanity.com/paper/1911.09032/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 30 of 36)

- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [Detection of out-of-distribution samples using binary neuron activation patterns](https://mlsanity.com/paper/2212.14268/) (2022)
- [Benchmarking Safety Monitors for Image Classifiers with Machine Learning](https://mlsanity.com/paper/2110.01232/) (2021)
- [Beyond Robustness: A Taxonomy of Approaches towards Resilient Multi-Robot Systems](https://mlsanity.com/paper/2109.12343/) (2021)
- [Correct-by-Construction Runtime Enforcement in AI -- A Survey](https://mlsanity.com/paper/2208.14426/) (2022)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Runtime Monitoring for Out-of-Distribution Detection in Object Detection Neural Networks](https://mlsanity.com/paper/2212.07773/) (2022)
- [Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes](https://mlsanity.com/paper/2104.14435/) (2021)
- [SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networks](https://mlsanity.com/paper/2106.01917/) (2021)
- [What, Indeed, is an Achievable Provable Guarantee for Learning-Enabled Safety Critical Systems](https://mlsanity.com/paper/2307.11784/) (2023)
- [A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networks](https://mlsanity.com/paper/2310.20349/) (2023)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [BAM: Box Abstraction Monitors for Real-time OoD Detection in Object Detection](https://mlsanity.com/paper/2403.18373/) (2024)
- [Monitoring Object Detection Abnormalities via Data-Label and Post-Algorithm Abstractions](https://mlsanity.com/paper/2103.15456/) (2021)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Prioritizing Corners in OoD Detectors via Symbolic String Manipulation](https://mlsanity.com/paper/2205.07736/) (2022)
- [Safety Monitoring of Machine Learning Perception Functions: a Survey](https://mlsanity.com/paper/2412.06869/) (2024)
- [Can we Defend Against the Unknown? An Empirical Study About Threshold Selection for Neural Network Monitoring](https://mlsanity.com/paper/2405.08654/) (2024)
- [Monitizer: Automating Design and Evaluation of Neural Network Monitors](https://mlsanity.com/paper/2405.10350/) (2024)
- [Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forces](https://mlsanity.com/paper/2410.06051/) (2024)
- [Edge Case Detection in Automated Driving: Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2410.08491/) (2024)
- [Runtime Safety Monitoring of Deep Neural Networks for Perception: A Survey](https://mlsanity.com/paper/2511.05982/) (2025)
- [Abstraction and Symbolic Execution of Deep Neural Networks with Bayesian Approximation of Hidden Features](https://mlsanity.com/paper/2103.03704/) (2021)
- [Runtime Monitoring DNN-Based Perception](https://mlsanity.com/paper/2310.03999/) (2023)
- [A New Perspective On AI Safety Through Control Theory Methodologies](https://mlsanity.com/paper/2506.23703/) (2025)
- [Out-of-Distribution Detection using Counterfactual Distance](https://mlsanity.com/paper/2508.10148/) (2025)
- [Hack The Box: Fooling Deep Learning Abstraction-Based Monitors](https://mlsanity.com/paper/2107.04764/) (2021)
- [Preventing Errors in Person Detection: A Part-Based Self-Monitoring Framework](https://mlsanity.com/paper/2307.04533/) (2023)
- [On the Detection of Anomalous or Out-Of-Distribution Data in Vision Models Using Statistical Techniques](https://mlsanity.com/paper/2403.15497/) (2024)

## Abstract (arXiv)

Neural networks have demonstrated unmatched performance in a range of classification tasks. Despite numerous efforts of the research community, novelty detection remains one of the significant limitations of neural networks. The ability to identify previously unseen inputs as novel is crucial for our understanding of the decisions made by neural networks. At runtime, inputs not falling into any of the categories learned during training cannot be classified correctly by the neural network. Existing approaches treat the neural network as a black box and try to detect novel inputs based on the confidence of the output predictions. However, neural networks are not trained to reduce their confidence for novel inputs, which limits the effectiveness of these approaches. We propose a framework to monitor a neural network by observing the hidden layers. We employ a common abstraction from program analysis - boxes - to identify novel behaviors in the monitored layers, i.e., inputs that cause behaviors outside the box. For each neuron, the boxes range over the values seen in training. The framework is efficient and flexible to achieve a desired trade-off between raising false warnings and detecting novel inputs. We illustrate the performance and the robustness to variability in the unknown classes on popular image-classification benchmarks.