A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions
Not on the map. Why?
Abstract (arXiv)
We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public .
Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 13 of its 16 references that are in the map. First posted on arXiv 17 May 2026.
Builds on
13 of 16 references.
References · 13
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 849
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 149
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
- 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
- 2024The Super Weight in Large Language Modelscited by 51
- 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
- 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
- 2024From Attention to Activation: Unravelling the Enigmas of Large Language Modelscited by 13
- 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
1 more
Built on by
Citing papers per year: 2026: 3.
3 papers.