mlsanityData to 14 Oct 2026

AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

arXiv 2506.14562 ↗First posted 17 Jun 2025cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Weight decay is a standard regularization technique for training large language models (LLMs). While it is common to assign a uniform decay rate to every layer, this approach overlooks the structural diversity of LLMs and the varying spectral properties across modules. In this paper, we introduce AlphaDecay, a simple yet effective method that adaptively assigns different weight decay strengths to each module of an LLM. Our approach is guided by Heavy-Tailed Self-Regularization (HT-SR) theory, which analyzes the empirical spectral density (ESD) of weight correlation matrices to quantify "heavy-tailedness." Modules exhibiting more pronounced heavy-tailed ESDs, reflecting stronger feature learning, are assigned weaker decay, while modules with lighter-tailed spectra receive stronger decay. Our method leverages tailored weight decay assignments to balance the module-wise differences in spectral properties, leading to improved performance. Extensive pre-training tasks with various model sizes from 60M to 1B demonstrate that AlphaDecay achieves better perplexity and generalization than conventional uniform decay and other adaptive decay baselines. Our code is available at https://github.com/hed-ucas/AlphaDecay.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

36 of 49 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  10. 2018Sparsified SGD with Memorycited by 376
  11. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
24 more
  1. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  2. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  3. 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
  4. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  5. 2024Make Your LLM Fully Utilize the Contextcited by 119
  6. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  7. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  8. 2024LLM Dataset Inference: Did you train on my dataset?cited by 110
  9. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  10. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  11. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  12. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  13. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 36
  14. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  15. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  16. 2024Weight decay induces low-rank attention layerscited by 27
  17. 2020Rethinking the Value of Transformer Componentscited by 25
  18. 2020On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspectivecited by 25
  19. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  20. 2019Adaptive Weight Decay for Deep Neural Networkscited by 13
  21. 2022Improving Robustness with Adaptive Weight Decaycited by 13
  22. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  23. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  24. 2024Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  2. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 7
  3. 2026A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributionscited by 3
  4. 2026One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMscited by 3
  5. 2026Layer Collapse in Diffusion Language Modelscited by 2
  6. 2026ActTail: Global Activation Sparsity in Large Language Modelscited by 1
  7. 2026Learning to Forget: Continual Learning with Adaptive Weight Decaycited by 1
  8. 2026Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networkscited by 1
  9. 2026Muse: Representation Geometry of Muon Beyond Normalized Momentumcited by 1
  10. 2026AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Trainingcited by 0
  11. 2026OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patternscited by 0
  12. 2026Stochastic Rounding Increases Small Singular Valuescited by 0
1 more
  1. 2026Bigger Text Encoders Can Hurt CLIP Zero-Shot Performancecited by 0