# Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

arXiv: https://arxiv.org/abs/2606.19367 · HTML: https://mlsanity.com/paper/2606.19367/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 8 of 8)

- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks](https://mlsanity.com/paper/2305.17212/) (2023)
- [Weight Decay may matter more than muP for Learning Rate Transfer in Practice](https://mlsanity.com/paper/2510.19093/) (2025)
- [Feature learning as alignment: a structural property of gradient descent in non-linear neural networks](https://mlsanity.com/paper/2402.05271/) (2024)
- [Robust Layerwise Scaling Rules by Proper Weight Decay Tuning](https://mlsanity.com/paper/2510.15262/) (2025)
- [Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?](https://mlsanity.com/paper/2410.14602/) (2024)
- [A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions](https://mlsanity.com/paper/2605.18898/) (2026)

## Built on by (most cited first, 2 of 2)

- [Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training](https://mlsanity.com/paper/2608.23573/) (2026)
- [A Mesoscopic View of Transformer Weights Through Row and Column Scale Fields](https://mlsanity.com/paper/2609.35852/) (2026)

## Abstract (arXiv)

Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter λ grows, overshoots, and then relaxes during AdamW training. We derive a leading-order three-force decomposition of the squared weight norm from the AdamW update: an alignment force measuring the correlation between weights and the adaptive update direction, an injection force from adaptive step magnitude, and a decay force from decoupled weight decay. On self-trained Pythia-70M models with ground-truth optimizer moments, alignment dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds and remaining robust to super-weight removal. Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation. These force dynamics directly govern the squared-norm component underlying λ(t); the remaining RMS-to-Weibull reconstruction offset is measurable and decomposes into bridge and integration components, totaling approximately 5-6% in densely sampled regions. To extend the analysis to real models where optimizer moments are unavailable, we introduce a spline displacement method that recovers the alignment force from sparse checkpoints with approximately 92-94% accuracy, about twice the naive two-point baseline. We further observe that the peak value of λ(t) varies with training-data coherence in our experiments, suggesting a data-dependent component of weight-scale growth that we leave to a controlled follow-up study. Code and data are available at https://github.com/tiexinding/NPM-Weibull-public.