mlsanityData to 14 Oct 2026

Estimating the Local Learning Coefficient at Scale

arXiv 2402.03698 ↗First posted 6 Feb 2024cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

The textitlocal learning coefficient (LLC) is a principled way of quantifying model complexity, originally derived in the context of Bayesian statistics using singular learning theory (SLT). Several methods are known for numerically estimating the local learning coefficient, but so far these methods have not been extended to the scale of modern deep learning architectures or data sets. Using a method developed in tt arXiv:2308.12108 [stat.ML] we empirically show how the LLC may be measured accurately and self-consistently for deep linear networks (DLNs) up to 100M parameters. We also show that the estimated LLC has the rescaling invariance that holds for the theoretical quantity.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 21 of its 55 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

21 of 55 references.

References · 21

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2020What is the State of Neural Network Pruning?cited by 583
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  7. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  8. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  9. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  12. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
9 more
  1. 2020Deep Learning is Singular, and That's Goodcited by 49
  2. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  3. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  4. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
  5. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
  6. 2019Estimating Real Log Canonical Thresholdscited by 9
  7. 2023Charting the Topography of the Neural Network Landscape with Thermal-Like Noisecited by 6
  8. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 6
  9. 2019On the overestimation of widely applicable Bayesian information criterioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 3.

6 papers.

  1. 2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
  2. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  3. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  4. 2024Upper Bound of Bayesian Generalization Error in Partial Concept Bottleneck Model (CBM): Partial CBM outperforms naive CBMcited by 1
  5. 2026Langevin dynamics along the zero set of real-analytic potentialscited by 0
  6. 2026Symmetries and Singularitiescited by 0