# DNN or k-NN: That is the Generalize vs. Memorize Question

arXiv: https://arxiv.org/abs/1805.06822 · HTML: https://mlsanity.com/paper/1805.06822/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2018.

## Built on by (most cited first, 16 of 16)

- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Reverse Engineering Self-Supervised Learning](https://mlsanity.com/paper/2305.15614/) (2023)
- [The Impact of Reinitialization on Generalization in Convolutional Neural Networks](https://mlsanity.com/paper/2109.00267/) (2021)
- [ResMem: Learn what you can and memorize the rest](https://mlsanity.com/paper/2302.01576/) (2023)
- [On the Implicit Bias Towards Minimal Depth of Deep Neural Networks](https://mlsanity.com/paper/2202.09028/) (2022)
- [The Curious Case of Benign Memorization](https://mlsanity.com/paper/2210.14019/) (2022)
- [Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks](https://mlsanity.com/paper/2408.04965/) (2024)
- [Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise](https://mlsanity.com/paper/2406.00396/) (2024)
- [Layer-Stack Temperature Scaling](https://mlsanity.com/paper/2211.10193/) (2022)
- [Exploring Learned Representations of Neural Networks with Principal Component Analysis](https://mlsanity.com/paper/2309.15328/) (2023)
- [On the Role of Neural Collapse in Meta Learning Models for Few-shot Learning](https://mlsanity.com/paper/2310.00451/) (2023)
- [kNet: A Deep kNN Network To Handle Label Noise](https://mlsanity.com/paper/2107.09735/) (2021)
- [On information captured by neural networks: connections with memorization and generalization](https://mlsanity.com/paper/2306.15918/) (2023)
- [Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes](https://mlsanity.com/paper/2308.00858/) (2023)
- [Least Squares Maximum and Weighted Generalization-Memorization Machines](https://mlsanity.com/paper/2308.16456/) (2023)

## Abstract (arXiv)

This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with new insights as to the ability of neural networks to both memorize the training data and generalize to new data at the same time, where k-NN serves as the ideal estimator that perfectly memorizes the data. We show that memorization of non-generalizing networks happens only at the last layers. Moreover, the behavior of DNNs compared to the linear classifiers SVM and LR is quite the same on the training and test data regardless of whether the network generalizes. On the other hand, the similarity to k-NN holds only at the absence of overfitting. Our results suggests that k-NN behavior of the network on new data is a sign of generalization. Moreover, it shows that memorization and generalization, which are traditionally considered to be contradicting to each other, are compatible and complementary.