# Detecting Memorization in ReLU Networks

arXiv: https://arxiv.org/abs/1810.03372 · HTML: https://mlsanity.com/paper/1810.03372/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2018.

## Built on by (most cited first, 1 of 1)

- [Universal Solutions of Feedforward ReLU Networks for Interpolations](https://mlsanity.com/paper/2208.07498/) (2022)

## Abstract (arXiv)

We propose a new notion of `non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our results demonstrate that as an indicator for memorization, our technique can be used to perform early stopping.