# The Local Learning Coefficient: A Singularity-Aware Complexity Measure

arXiv: https://arxiv.org/abs/2308.12108 · HTML: https://mlsanity.com/paper/2308.12108/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [Gradient descent aligns the layers of deep linear networks](https://mlsanity.com/paper/1810.02032/) (2018)
- [SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics](https://mlsanity.com/paper/2302.11055/) (2023)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Shortformer: Better Language Modeling using Shorter Inputs](https://mlsanity.com/paper/2012.15832/) (2020)
- [Deep Learning is Singular, and That's Good](https://mlsanity.com/paper/2010.11560/) (2020)
- [Why neural networks find simple solutions: the many regularizers of geometric complexity](https://mlsanity.com/paper/2209.13083/) (2022)
- [Minimum Description Length Revisited](https://mlsanity.com/paper/1908.08484/) (2019)
- [Incremental Learning in Diagonal Linear Networks](https://mlsanity.com/paper/2208.14673/) (2022)
- [Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning](https://mlsanity.com/paper/1803.01927/) (2018)
- [How degenerate is the parametrization of neural networks with the ReLU activation function?](https://mlsanity.com/paper/1905.09803/) (2019)
- [Estimating Real Log Canonical Thresholds](https://mlsanity.com/paper/1906.01341/) (2019)
- [Charting the Topography of the Neural Network Landscape with Thermal-Like Noise](https://mlsanity.com/paper/2304.01335/) (2023)
- [Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks](https://mlsanity.com/paper/2305.05089/) (2023)
- [On the overestimation of widely applicable Bayesian information criterion](https://mlsanity.com/paper/1908.10572/) (2019)
- [Learning Capacity: A Measure of the Effective Dimensionality of a Model](https://mlsanity.com/paper/2305.17332/) (2023)

## Built on by (most cited first, 30 of 51)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Loss Landscape Degeneracy and Stagewise Development in Transformers](https://mlsanity.com/paper/2402.02364/) (2024)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [An alignment safety case sketch based on debate](https://mlsanity.com/paper/2505.03989/) (2025)
- [Dynamics of Transient Structure in In-Context Linear Regression Transformers](https://mlsanity.com/paper/2501.17745/) (2025)
- [Bayesian Influence Functions for Hessian-Free Data Attribution](https://mlsanity.com/paper/2509.26544/) (2025)
- [Using Degeneracy in the Loss Landscape for Mechanistic Interpretability](https://mlsanity.com/paper/2405.10927/) (2024)
- [The Loss Kernel: A Geometric Probe for Deep Learning Interpretability](https://mlsanity.com/paper/2509.26537/) (2025)
- [Evolution of Concepts in Language Model Pre-Training](https://mlsanity.com/paper/2509.17196/) (2025)
- [Influence Dynamics and Stagewise Data Attribution](https://mlsanity.com/paper/2510.12071/) (2025)
- [Patterning: The Dual of Interpretability](https://mlsanity.com/paper/2601.13548/) (2026)
- [Estimating the Local Learning Coefficient at Scale](https://mlsanity.com/paper/2402.03698/) (2024)
- [Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities](https://mlsanity.com/paper/2602.00693/) (2026)
- [A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints](https://mlsanity.com/paper/2410.05612/) (2024)
- [Identifying Sparsely Active Circuits Through Local Loss Landscape Decomposition](https://mlsanity.com/paper/2504.00194/) (2025)
- [From Global to Local: A Scalable Benchmark for Local Posterior Sampling](https://mlsanity.com/paper/2507.21449/) (2025)
- [Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory](https://mlsanity.com/paper/2510.12077/) (2025)
- [Stagewise Reinforcement Learning and the Geometry of the Regret Landscape](https://mlsanity.com/paper/2601.07524/) (2026)
- [A Basin-Selection Perspective on Grokking via Singular Learning Theory](https://mlsanity.com/paper/2603.01192/) (2026)
- [From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability](https://mlsanity.com/paper/2603.29805/) (2026)
- [Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture](https://mlsanity.com/paper/2412.15113/) (2024)
- [Estimating the Probability of Sampling a Trained Neural Network at Random](https://mlsanity.com/paper/2501.18812/) (2025)
- [Modes of Sequence Models and Learning Coefficients](https://mlsanity.com/paper/2504.18048/) (2025)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)
- [Grokking vs. Learning: Same Features, Different Encodings](https://mlsanity.com/paper/2502.01739/) (2025)
- [Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2503.22478/) (2025)
- [Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning](https://mlsanity.com/paper/2605.07980/) (2026)
- [Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity](https://mlsanity.com/paper/2606.21158/) (2026)
- [Emergence of Computational Structure in a Neural Network Physics Simulator](https://mlsanity.com/paper/2504.11830/) (2025)
- [Scaling Laws for Uncertainty in Deep Learning](https://mlsanity.com/paper/2506.09648/) (2025)

## Abstract (arXiv)

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.