# Identity Crisis: Memorization and Generalization under Extreme Overparameterization

arXiv: https://arxiv.org/abs/1902.04698 · HTML: https://mlsanity.com/paper/1902.04698/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2019.

## Built on by (most cited first, 30 of 33)

- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Properties of the After Kernel](https://mlsanity.com/paper/2105.10585/) (2021)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Memorization and Generalization in Neural Code Intelligence Models](https://mlsanity.com/paper/2106.08704/) (2021)
- [The Pitfalls of Memorization: When Memorization Hurts Generalization](https://mlsanity.com/paper/2412.07684/) (2024)
- [Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm](https://mlsanity.com/paper/2102.12238/) (2021)
- [Memorization in deep learning: A survey](https://mlsanity.com/paper/2406.03880/) (2024)
- [Analysis of feature learning in weight-tied autoencoders via the mean field lens](https://mlsanity.com/paper/2102.08373/) (2021)
- [ReSup: Reliable Label Noise Suppression for Facial Expression Recognition](https://mlsanity.com/paper/2305.17895/) (2023)
- [The surprising impact of mask-head architecture on novel class segmentation](https://mlsanity.com/paper/2104.00613/) (2021)
- [SinIR: Efficient General Image Manipulation with Single Image Reconstruction](https://mlsanity.com/paper/2106.07140/) (2021)
- [Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD](https://mlsanity.com/paper/2307.00310/) (2023)
- [Navigating the Latent Space Dynamics of Neural Models](https://mlsanity.com/paper/2505.22785/) (2025)
- [What do larger image classifiers memorise?](https://mlsanity.com/paper/2310.05337/) (2023)
- [Mini but Mighty: Finetuning ViTs with Mini Adapters](https://mlsanity.com/paper/2311.03873/) (2023)
- [Trade-offs in Data Memorization via Strong Data Processing Inequalities](https://mlsanity.com/paper/2506.01855/) (2025)
- [A Mechanism for Producing Aligned Latent Spaces with Autoencoders](https://mlsanity.com/paper/2106.15456/) (2021)
- [SpanSeq: Similarity-based sequence data splitting method for improved development and assessment of deep learning projects](https://mlsanity.com/paper/2402.14482/) (2024)
- [Do autoencoders need a bottleneck for anomaly detection?](https://mlsanity.com/paper/2202.12637/) (2022)
- [Training Dynamics of Deep Network Linear Regions](https://mlsanity.com/paper/2310.12977/) (2023)
- [Frequency Dropout: Feature-Level Regularization via Randomized Filtering](https://mlsanity.com/paper/2209.09844/) (2022)
- [Anonymous Learning via Look-Alike Clustering: A Precise Analysis of Model Generalization](https://mlsanity.com/paper/2310.04015/) (2023)
- [Feature Importance in a Deep Learning Climate Emulator](https://mlsanity.com/paper/2108.13203/) (2021)
- [Stretched and measured neural predictions of complex network dynamics](https://mlsanity.com/paper/2301.04900/) (2023)
- [Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learning](https://mlsanity.com/paper/2306.09991/) (2023)
- [Laying Anchors: Semantically Priming Numerals in Language Modeling](https://mlsanity.com/paper/2404.01536/) (2024)
- [Hidden Activations Are Not Enough: A General Approach to Neural Network Predictions](https://mlsanity.com/paper/2409.13163/) (2024)
- [Learned Random Label Predictions as a Neural Network Complexity Metric](https://mlsanity.com/paper/2411.19640/) (2024)
- [Improving the Robustness of Neural Multiplication Units with Reversible Stochasticity](https://mlsanity.com/paper/2211.05624/) (2022)
- [CRAFT: Contextual Re-Activation of Filters for face recognition Training](https://mlsanity.com/paper/2312.00072/) (2023)

## Abstract (arXiv)

We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identity-mapping task. We examine fully-connected and convolutional networks (FCN and CNN), both linear and nonlinear, initialized randomly and then trained to minimize the reconstruction error. The trained networks stereotypically take one of two forms: the constant function (memorization) and the identity function (generalization). We formally characterize generalization in single-layer FCNs and CNNs. We show empirically that different architectures exhibit strikingly different inductive biases. For example, CNNs of up to 10 layers are able to generalize from a single example, whereas FCNs cannot learn the identity function reliably from 60k examples. Deeper CNNs often fail, but nonetheless do astonishing work to memorize the training output: because CNN biases are location invariant, the model must progressively grow an output pattern from the image boundaries via the coordination of many layers. Our work helps to quantify and visualize the sensitivity of inductive biases to architectural choices such as depth, kernel width, and number of channels.