mlsanityData to 7 Oct 2026

A Two-Scale Complexity Measure for Deep Learning Models

arXiv 2401.09184 ↗First posted 17 Jan 2024stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce a novel capacity measure 2sED for statistical models based on the effective dimension. The new quantity provably bounds the generalization error under mild assumptions on the model. Furthermore, simulations on standard data sets and popular model architectures show that 2sED correlates well with the training error. For Markovian models, we show how to efficiently approximate 2sED from below through a layerwise iterative approach, which allows us to tackle deep learning models with a large number of parameters. Simulation results suggest that the approximation is good for different prominent models and data sets.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

15 of 32 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  3. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  4. 2020The power of quantum neural networkscited by 377
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  7. 2016On the Expressive Power of Deep Neural Networkscited by 302
  8. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  9. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  10. 2021Model Complexity of Deep Learning: A Surveycited by 64
  11. 2020Generalization bounds for deep learningcited by 38
  12. 2020A scale-dependent notion of effective dimensioncited by 17
3 more
  1. 2021Effective dimension of machine learning modelscited by 14
  2. 2019On the complexity of logistic regression modelscited by 3
  3. 2023Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problemscited by 0
  2. 2025More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problemscited by 0
  3. 2026Beyond NNGP: Large Deviations and Feature Learning in Bayesian Neural Networkscited by 0