# SETOL: A Semi-Empirical Theory of (Deep) Learning

arXiv: https://arxiv.org/abs/2507.17912 · HTML: https://mlsanity.com/paper/2507.17912/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2025.

## Built on by (most cited first, 12 of 12)

- [The Universal Weight Subspace Hypothesis](https://mlsanity.com/paper/2512.05117/) (2025)
- [HTMuon: Improving Muon via Heavy-Tailed Spectral Correction](https://mlsanity.com/paper/2603.10067/) (2026)
- [Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with Weightwatcher](https://mlsanity.com/paper/2602.02859/) (2026)
- [Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory](https://mlsanity.com/paper/2605.12394/) (2026)
- [Beyond Accuracy: A Unified Random Matrix Theory Diagnostic Framework for Crash Classification Models](https://mlsanity.com/paper/2602.19528/) (2026)
- [Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models](https://mlsanity.com/paper/2604.07363/) (2026)
- [Characterizing Learning in Deep Neural Networks using Tractable Algorithmic Complexity Analysis](https://mlsanity.com/paper/2605.15551/) (2026)
- [The Hamilton-Jacobi Theory of Deep Learning](https://mlsanity.com/paper/2605.28983/) (2026)
- [Fast Generalization after Interpolation via Critically Damped Momentum Optimization](https://mlsanity.com/paper/2606.01521/) (2026)
- [Spectral Signatures of Large Language Models](https://mlsanity.com/paper/2607.03377/) (2026)
- [Price Stability in the European Union: A Systemic Approach Using Random Matrix Theory](https://mlsanity.com/paper/2609.35011/) (2026)
- [Why Heavy-Tailed Weight Spectra Predict Model Quality](https://mlsanity.com/paper/2610.13123/) (2026)

## Abstract (arXiv)

We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenological theory of Heavy-Tailed Self-Regularization (HTSR): the heavy-tailed power-law layer quality metrics, alpha and alpha-hat. In prior work, these metrics have been shown to predict trends in the test accuracies of pretrained SOTA NN models, importantly, without needing access to either testing or training data. Our SETOL uses techniques from statistical mechanics as well as advanced methods from random matrix theory and quantum chemistry. The derivation suggests new mathematical preconditions for ideal learning, including a new metric, ERG, which is equivalent to applying a single step of the Wilson Exact Renormalization Group. We test the assumptions and predictions of SETOL on a simple 3-layer multilayer perceptron (MLP), demonstrating excellent agreement with the key theoretical assumptions. For SOTA NN models, we show how to estimate the individual layer qualities of a trained NN by simply computing the empirical spectral density (ESD) of the layer weight matrices and plugging this ESD into our SETOL formulas. Notably, we examine the performance of the HTSR alpha and the SETOL ERG layer quality metrics, and find that they align remarkably well, both on our MLP and on SOTA NNs.