# A Two-Scale Complexity Measure for Deep Learning Models

arXiv: https://arxiv.org/abs/2401.09184 · HTML: https://mlsanity.com/paper/2401.09184/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 17 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [The power of quantum neural networks](https://mlsanity.com/paper/2011.00027/) (2020)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Fisher-Rao Metric, Geometry, and Complexity of Neural Networks](https://mlsanity.com/paper/1711.01530/) (2017)
- [Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach](https://mlsanity.com/paper/1806.01316/) (2018)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [Generalization bounds for deep learning](https://mlsanity.com/paper/2012.04115/) (2020)
- [A scale-dependent notion of effective dimension](https://mlsanity.com/paper/2001.10872/) (2020)
- [Effective dimension of machine learning models](https://mlsanity.com/paper/2112.04807/) (2021)
- [On the complexity of logistic regression models](https://mlsanity.com/paper/1903.00386/) (2019)
- [Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers](https://mlsanity.com/paper/2303.16464/) (2023)

## Built on by (most cited first, 3 of 3)

- [Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems](https://mlsanity.com/paper/2503.13764/) (2025)
- [More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problems](https://mlsanity.com/paper/2505.02985/) (2025)
- [Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networks](https://mlsanity.com/paper/2602.22925/) (2026)

## Abstract (arXiv)

We introduce a novel capacity measure 2sED for statistical models based on the effective dimension. The new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layerwise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets.