# Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

arXiv: https://arxiv.org/abs/2608.23573 · HTML: https://mlsanity.com/paper/2608.23573/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 23 of 23)

- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks](https://mlsanity.com/paper/2305.17212/) (2023)
- [Neural networks trained with SGD learn distributions of increasing complexity](https://mlsanity.com/paper/2211.11567/) (2022)
- [How to set AdamW's weight decay as you scale model and dataset size](https://mlsanity.com/paper/2405.13698/) (2024)
- [Neural Networks Learn Statistics of Increasing Complexity](https://mlsanity.com/paper/2402.04362/) (2024)
- [Approaching Deep Learning through the Spectral Dynamics of Weights](https://mlsanity.com/paper/2408.11804/) (2024)
- [A distributional simplicity bias in the learning dynamics of transformers](https://mlsanity.com/paper/2410.19637/) (2024)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Robust Layerwise Scaling Rules by Proper Weight Decay Tuning](https://mlsanity.com/paper/2510.15262/) (2025)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?](https://mlsanity.com/paper/2410.14602/) (2024)
- [A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions](https://mlsanity.com/paper/2605.18898/) (2026)
- [Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics](https://mlsanity.com/paper/2606.19367/) (2026)
- [Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks](https://mlsanity.com/paper/2510.25480/) (2025)
- [Correction of Decoupled Weight Decay](https://mlsanity.com/paper/2512.08217/) (2025)
- [Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks](https://mlsanity.com/paper/2603.27885/) (2026)

## Built on by (most cited first, 1 of 1)

- [A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields](https://mlsanity.com/paper/2609.35852/) (2026)

## Abstract (arXiv)

A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape k ≈ 1.2 is stable across layers and models, so the scale λ carries most training-induced movement. What corpus property sets how much λ grows? Using the bigram conditional entropy D = H(next mid prev), a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, λ2 - λ02 = C0(η) + C1(η)(Hr - D)0.59, where Hr is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-η coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto (Hr - D)0.59 with unit slope (R2 = 0.941; direct per-η fits are weaker, R2 ≈ 0.82). Because D is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader Phi(D,R,A,H) data-to-weight framework.