# Subset Scanning Over Neural Network Activations

arXiv: https://arxiv.org/abs/1810.08676 · HTML: https://mlsanity.com/paper/1810.08676/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 2 of 2)

- [Efficient Representation of the Activation Space in Deep Neural Networks](https://mlsanity.com/paper/2312.08143/) (2023)
- [Machine Beats Machine: Machine Learning Models to Defend Against Adversarial Attacks](https://mlsanity.com/paper/2209.13963/) (2022)

## Abstract (arXiv)

This work views neural networks as data generating systems and applies anomalous pattern detection techniques on that data in order to detect when a network is processing an anomalous input. Detecting anomalies is a critical component for multiple machine learning problems including detecting adversarial noise. More broadly, this work is a step towards giving neural networks the ability to recognize an out-of-distribution sample. This is the first work to introduce "Subset Scanning" methods from the anomalous pattern detection domain to the task of detecting anomalous input of neural networks. Subset scanning treats the detection problem as a search for the most anomalous subset of node activations (i.e., highest scoring subset according to non-parametric scan statistics). Mathematical properties of these scoring functions allow the search to be completed in log-linear rather than exponential time while still guaranteeing the most anomalous subset of nodes in the network is identified for a given input. Quantitative results for detecting and characterizing adversarial noise are provided for CIFAR-10 images on a simple convolutional neural network. We observe an "interference" pattern where anomalous activations in shallow layers suppress the activation structure of the original image in deeper layers.