mlsanityData to 14 Oct 2026

The Local Learning Coefficient: A Singularity-Aware Complexity Measure

arXiv 2308.12108 ↗First posted 23 Aug 2023stat.ML cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 29 of its 73 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

29 of 73 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  6. 2017Exploring Generalization in Deep Learningcited by 600
  7. 2020What is the State of Neural Network Pruning?cited by 583
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  11. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  12. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
17 more
  1. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
  2. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  3. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  4. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  5. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  6. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  7. 2020Deep Learning is Singular, and That's Goodcited by 49
  8. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  9. 2019Minimum Description Length Revisitedcited by 37
  10. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  11. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
  12. 2019How degenerate is the parametrization of neural networks with the ReLU activation function?cited by 15
  13. 2019Estimating Real Log Canonical Thresholdscited by 9
  14. 2023Charting the Topography of the Neural Network Landscape with Thermal-Like Noisecited by 6
  15. 2023Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networkscited by 6
  16. 2019On the overestimation of widely applicable Bayesian information criterioncited by 4
  17. 2023Learning Capacity: A Measure of the Effective Dimensionality of a Modelcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 17; 2026: 25.

51 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  2. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  3. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 29
  4. 2025An alignment safety case sketch based on debatecited by 16
  5. 2025Dynamics of Transient Structure in In-Context Linear Regression Transformerscited by 14
  6. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 14
  7. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 13
  8. 2025The Loss Kernel: A Geometric Probe for Deep Learning Interpretabilitycited by 9
  9. 2025Evolution of Concepts in Language Model Pre-Trainingcited by 8
  10. 2025Influence Dynamics and Stagewise Data Attributioncited by 7
  11. 2026Patterning: The Dual of Interpretabilitycited by 7
  12. 2024Estimating the Local Learning Coefficient at Scalecited by 6
38 more
  1. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  2. 2024A Bayesian Model Selection Criterion for Selecting Pretraining Checkpointscited by 5
  3. 2025Identifying Sparsely Active Circuits Through Local Loss Landscape Decompositioncited by 5
  4. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  5. 2025Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theorycited by 5
  6. 2026Stagewise Reinforcement Learning and the Geometry of the Regret Landscapecited by 4
  7. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  8. 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 4
  9. 2024Associative memory inspires improvements for in-context learning using a novel attention residual stream architecturecited by 3
  10. 2025Estimating the Probability of Sampling a Trained Neural Network at Randomcited by 3
  11. 2025Modes of Sequence Models and Learning Coefficientscited by 3
  12. 2026Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scalecited by 3
  13. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  14. 2025Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descentcited by 2
  15. 2026Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learningcited by 2
  16. 2026Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexitycited by 2
  17. 2025Emergence of Computational Structure in a Neural Network Physics Simulatorcited by 1
  18. 2025Scaling Laws for Uncertainty in Deep Learningcited by 1
  19. 2026Statistical Learning Analysis of Physics-Informed Neural Networkscited by 1
  20. 2026The Geometric Anatomy of Capability Acquisition in Transformerscited by 1
  21. 2026Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networkscited by 1
  22. 2026Mechanistic Anomaly Detection via Functional Attributioncited by 1
  23. 2024NGD converges to less degenerate solutions than SGDcited by 0
  24. 2024Statistical inference for quantum singular modelscited by 0
  25. 2025Decomposing Prediction Mechanisms for In-Context Recallcited by 0
  26. 2025Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networkscited by 0
  27. 2026Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regimecited by 0
  28. 2026The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networkscited by 0
  29. 2026Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probescited by 0
  30. 2026Singular Learning and Occam's Razor in Deep Monomial Networkscited by 0
  31. 2026Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignmentcited by 0
  32. 2026Langevin dynamics along the zero set of real-analytic potentialscited by 0
  33. 2026The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verificationcited by 0
  34. 2026Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Modelscited by 0
  35. 2026Functional Degeneracy in Neural Networks: Measurement and Pruningcited by 0
  36. 2026Patterning in Practice: Debiasing Reward Models with Susceptibilitiescited by 0
  37. 2026Symmetries and Singularitiescited by 0
  38. 2026Math for AI safety: an invitation for mathematicianscited by 0

50 most cited of 51 All 51 (JSON)