Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training
Not on the map. Why?
Abstract (arXiv)
A trained transformer's weight magnitudes can be summarized by a two-parameter Weibull distribution whose shape k ≈ 1.2 is stable across layers and models, so the scale λ carries most training-induced movement. What corpus property sets how much λ grows? Using the bigram conditional entropy D = H(next mid prev), a training-free statistic computed before training, we find across controlled corruption families a learning-rate-conditioned law, λ2 - λ02 = C0(η) + C1(η)(Hr - D)0.59, where Hr is a matched-budget shuffle baseline. The convex exponent is inherited from an independently measured data-side saturation relation rather than fitted directly to the growth curve. After removing the two per-η coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto (Hr - D)0.59 with unit slope (R2 = 0.941; direct per-η fits are weaker, R2 ≈ 0.82). Because D is computed before training, the law is a forward predictor: an end-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The readout holds at model and per-layer resolutions and across two tested architectures, with the functional form preserved and only the coefficients changing. It also marks its boundary: cross-corpus prediction over-predicts code, implicating redundancy as a second axis of a broader Phi(D,R,A,H) data-to-weight framework.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 27 Jun 2026.
Builds on
23 of 25 references.
References · 23
- 2020GLU Variants Improve Transformercited by 1,759
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,049
- 2016Language Modeling with Gated Convolutional Networkscited by 694
- 2023Scaling Data-Constrained Language Modelscited by 398
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 278
- 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
- 2017L2 Regularization versus Batch and Weight Normalizationcited by 123
- 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 52
- 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
- 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
- 2024Neural Networks Learn Statistics of Increasing Complexitycited by 30
- 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 30
11 more
- 2024A distributional simplicity bias in the learning dynamics of transformerscited by 24
- 2024Loss-to-Loss Prediction: Scaling Laws for All Datasetscited by 15
- 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
- 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
- 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
- 2024Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?cited by 4
- 2026A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributionscited by 3
- 2026Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamicscited by 2
- 2025Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Taskscited by 1
- 2025Correction of Decoupled Weight Decaycited by 1
- 2026Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networkscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.