# Relating Regularization and Generalization through the Intrinsic Dimension of Activations

arXiv: https://arxiv.org/abs/2211.13239 · HTML: https://mlsanity.com/paper/2211.13239/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Estimating the intrinsic dimension of datasets by a minimal neighborhood information](https://mlsanity.com/paper/1803.06992/) (2018)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [The Description Length of Deep Learning Models](https://mlsanity.com/paper/1802.07044/) (2018)
- [Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks](https://mlsanity.com/paper/2111.13171/) (2021)
- [A Topological Regularizer for Classifiers via Persistent Homology](https://mlsanity.com/paper/1806.10714/) (2018)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Dimensionality compression and expansion in Deep Neural Networks](https://mlsanity.com/paper/1906.00443/) (2019)
- [LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood](https://mlsanity.com/paper/2206.14882/) (2022)
- [LDMNet: Low Dimensional Manifold Regularized Neural Networks](https://mlsanity.com/paper/1711.06246/) (2017)
- [Intrinsic Dimension Estimation Using Wasserstein Distances](https://mlsanity.com/paper/2106.04018/) (2021)
- [Geometry-Aware Maximum Likelihood Estimation of Intrinsic Dimension](https://mlsanity.com/paper/1904.06151/) (2019)
- [Degrees of Freedom in Deep Neural Networks](https://mlsanity.com/paper/1603.09260/) (2016)
- [Tangent Space and Dimension Estimation with the Wasserstein Distance](https://mlsanity.com/paper/2110.06357/) (2021)
- [Unsupervised Embedding Adaptation via Early-Stage Feature Reconstruction for Few-Shot Classification](https://mlsanity.com/paper/2106.11486/) (2021)
- [CoopSubNet: Cooperating Subnetwork for Data-Driven Regularization of Deep Networks under Limited Training Budgets](https://mlsanity.com/paper/1906.05441/) (2019)
- [The Foes of Neural Network's Data Efficiency Among Unnecessary Input Dimensions](https://mlsanity.com/paper/2107.06409/) (2021)

## Built on by (most cited first, 9 of 9)

- [A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models](https://mlsanity.com/paper/2406.03537/) (2024)
- [Deep Regression Representation Learning with Topology](https://mlsanity.com/paper/2404.13904/) (2024)
- [Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations](https://mlsanity.com/paper/2406.15812/) (2024)
- [On Convolutions, Intrinsic Dimension, and Diffusion Models](https://mlsanity.com/paper/2506.20705/) (2025)
- [Topological Signatures of Grokking](https://mlsanity.com/paper/2605.06352/) (2026)
- [At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics](https://mlsanity.com/paper/2607.06639/) (2026)
- [HomID : Benchmarking Intrinsic Dimension Estimators on Homogenous Manifolds with Anisotropic Embeddings](https://mlsanity.com/paper/2510.01335/) (2025)
- [Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation](https://mlsanity.com/paper/2511.14981/) (2025)
- [Higher Structures in Deep Learning](https://mlsanity.com/paper/2609.00472/) (2026)

## Abstract (arXiv)

Given a pair of models with similar training set performance, it is natural to assume that the model that possesses simpler internal representations would exhibit better generalization. In this work, we provide empirical evidence for this intuition through an analysis of the intrinsic dimension (ID) of model activations, which can be thought of as the minimal number of factors of variation in the model's representation of the data. First, we show that common regularization techniques uniformly decrease the last-layer ID (LLID) of validation set activations for image classification models and show how this strongly affects generalization performance. We also investigate how excessive regularization decreases a model's ability to extract features from data in earlier layers, leading to a negative effect on validation accuracy even while LLID continues to decrease and training accuracy remains near-perfect. Finally, we examine the LLID over the course of training of models that exhibit grokking. We observe that well after training accuracy saturates, when models ``grok'' and validation accuracy suddenly improves from random to perfect, there is a co-occurent sudden drop in LLID, thus providing more insight into the dynamics of sudden generalization.