# Scaling Laws are Redundancy Laws

arXiv: https://arxiv.org/abs/2509.20721 · HTML: https://mlsanity.com/paper/2509.20721/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 14 of 14)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Scaling Laws For Diffusion Transformers](https://mlsanity.com/paper/2410.08184/) (2024)
- [Scaling Laws for Deep Learning](https://mlsanity.com/paper/2108.07686/) (2021)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory](https://mlsanity.com/paper/2405.08707/) (2024)
- [An Information-Theoretic Approach to Generalization Theory](https://mlsanity.com/paper/2408.13275/) (2024)
- [How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines](https://mlsanity.com/paper/2502.12051/) (2025)

## Built on by (most cited first, 3 of 3)

- [Scaling Laws and In-Context Learning: A Unified Theoretical Framework](https://mlsanity.com/paper/2511.06232/) (2025)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models](https://mlsanity.com/paper/2604.27115/) (2026)

## Abstract (arXiv)

Scaling laws, a defining feature of deep learning, reveal a striking power-law improvement in model performance with increasing dataset and model size. Yet, their mathematical origins, especially the scaling exponent, have remained elusive. In this work, we show that scaling laws can be formally explained as redundancy laws. Using kernel regression, we show that a polynomial tail in the data covariance spectrum yields an excess risk power law with exponent alpha = 2s / (2s + 1/beta), where beta controls the spectral tail and 1/beta measures redundancy. This reveals that the learning curve's slope is not universal but depends on data redundancy, with steeper spectra accelerating returns to scale. We establish the law's universality across boundedly invertible transformations, multi-modal mixtures, finite-width approximations, and Transformer architectures in both linearized (NTK) and feature-learning regimes. This work delivers the first rigorous mathematical explanation of scaling laws as finite-sample redundancy laws, unifying empirical observations with theoretical foundations.