# Effective dimension of machine learning models

arXiv: https://arxiv.org/abs/2112.04807 · HTML: https://mlsanity.com/paper/2112.04807/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [The power of quantum neural networks](https://mlsanity.com/paper/2011.00027/) (2020)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [Rademacher Complexity for Adversarially Robust Generalization](https://mlsanity.com/paper/1810.11914/) (2018)
- [Fisher-Rao Metric, Geometry, and Complexity of Neural Networks](https://mlsanity.com/paper/1711.01530/) (2017)
- [Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning](https://mlsanity.com/paper/1912.08881/) (2019)
- [A scale-dependent notion of effective dimension](https://mlsanity.com/paper/2001.10872/) (2020)
- [Identifying Generalization Properties in Neural Networks](https://mlsanity.com/paper/1809.07402/) (2018)
- [Approximate Fisher Information Matrix to Characterise the Training of Deep Neural Networks](https://mlsanity.com/paper/1810.06767/) (2018)
- [Compact and Computationally Efficient Representation of Deep Neural Networks](https://mlsanity.com/paper/1805.10692/) (2018)

## Built on by (most cited first, 14 of 14)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Do Quantum Circuit Born Machines Generalize?](https://mlsanity.com/paper/2207.13645/) (2022)
- [Generalization of Quantum Machine Learning Models Using Quantum Fisher Information Metric](https://mlsanity.com/paper/2303.13462/) (2023)
- [Understanding Generalization in Quantum Machine Learning with Margins](https://mlsanity.com/paper/2411.06919/) (2024)
- [Quantum neural networks force fields generation](https://mlsanity.com/paper/2203.04666/) (2022)
- [Quantum Machine Learning for Malware Classification](https://mlsanity.com/paper/2305.09674/) (2023)
- [A Two-Scale Complexity Measure for Deep Learning Models](https://mlsanity.com/paper/2401.09184/) (2024)
- [Generalization Bounds in Hybrid Quantum-Classical Machine Learning Models](https://mlsanity.com/paper/2504.08456/) (2025)
- [A universal compression theory for lottery ticket hypothesis and neural scaling laws](https://mlsanity.com/paper/2510.00504/) (2025)
- [Data-Dependent Generalization Bounds for Parameterized Quantum Models Under Noise](https://mlsanity.com/paper/2412.11451/) (2024)
- [Quantum feature-map learning with reduced resource overhead](https://mlsanity.com/paper/2510.03389/) (2025)
- [Fisher Information, Training and Bias in Fourier Regression Models](https://mlsanity.com/paper/2510.06945/) (2025)
- [Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI](https://mlsanity.com/paper/2603.25033/) (2026)
- [Signal-Noise Factorization Isolates Nuisance Variation into Removable Subspaces](https://mlsanity.com/paper/2610.00751/) (2026)

## Abstract (arXiv)

Making statements about the performance of trained models on tasks involving new data is one of the primary goals of machine learning, i.e., to understand the generalization power of a model. Various capacity measures try to capture this ability, but usually fall short in explaining important characteristics of models that we observe in practice. In this study, we propose the local effective dimension as a capacity measure which seems to correlate well with generalization error on standard data sets. Importantly, we prove that the local effective dimension bounds the generalization error and discuss the aptness of this capacity measure for machine learning models.