# Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation

arXiv: https://arxiv.org/abs/2112.08798 · HTML: https://mlsanity.com/paper/2112.08798/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?](https://mlsanity.com/paper/2012.06421/) (2020)
- [BackPACK: Packing more into backprop](https://mlsanity.com/paper/1912.10985/) (2019)
- [What Do Neural Networks Learn When Trained With Random Labels?](https://mlsanity.com/paper/2006.10455/) (2020)
- [Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization](https://mlsanity.com/paper/2002.10657/) (2020)
- [Efficient Estimation of Influence of a Training Instance](https://mlsanity.com/paper/2012.04207/) (2020)
- [Neural Network Memorization Dissection](https://mlsanity.com/paper/1911.09537/) (2019)

## Built on by (most cited first, 7 of 7)

- [Dataset Difficulty and the Role of Inductive Bias](https://mlsanity.com/paper/2401.01867/) (2024)
- [Range Membership Inference Attacks](https://mlsanity.com/paper/2408.05131/) (2024)
- [Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond](https://mlsanity.com/paper/2503.07501/) (2025)
- [Measuring Unintended Memorisation of Unique Private Features in Neural Networks](https://mlsanity.com/paper/2202.08099/) (2022)
- [Influence Scores at Scale for Efficient Language Data Sampling](https://mlsanity.com/paper/2311.16298/) (2023)
- [WaKA: Data Attribution using K-Nearest Neighbors and Membership Privacy Principles](https://mlsanity.com/paper/2411.01357/) (2024)
- [Unintended memorisation of unique features in neural networks](https://mlsanity.com/paper/2205.10079/) (2022)

## Abstract (arXiv)

Over-parameterized deep neural networks are able to achieve excellent training accuracy while maintaining a small generalization error. It has also been found that they are able to fit arbitrary labels, and this behaviour is referred to as the phenomenon of memorization. In this work, we study the phenomenon of memorization with turn-over dropout, an efficient method to estimate influence and memorization, for data with true labels (real data) and data with random labels (random data). Our main findings are: (i) For both real data and random data, the optimization of easy examples (e.g., real data) and difficult examples (e.g., random data) are conducted by the network simultaneously, with easy ones at a higher speed; (ii) For real data, a correct difficult example in the training dataset is more informative than an easy one. By showing the existence of memorization on random data and real data, we highlight the consistency between them regarding optimization and we emphasize the implication of memorization during optimization.