mlsanityData to 14 Oct 2026

Variance Control via Weight Rescaling in LLM Pre-training

arXiv 2503.17500 ↗First posted 21 Mar 2025cs.LG cs.CL stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

The outcome of Large Language Model (LLM) pre-training strongly depends on weight initialization and variance control strategies. Although the importance of initial variance control has been well documented in neural networks in general, the literature on initialization and management of its growth during LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce the Layer Index Rescaling (LIR) weight initialization scheme, and the Target Variance Rescaling (TVR) variance control strategy. Experiments on a 1B parameter LLaMA model demonstrate that better variance management using these techniques yields substantial improvements in downstream task performance (up to 4.6% on common pre-training benchmarks) and reduces extreme activation values, thus mitigating challenges associated with quantization and low-precision training. Our code is available at: https://github.com/bluorion-com/weight_rescaling.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

23 of 39 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2020GLU Variants Improve Transformercited by 1,804
  11. 2019Root Mean Square Layer Normalizationcited by 1,480
  12. 2024Massive Activations in Large Language Modelscited by 294
11 more
  1. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  2. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  4. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  5. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  6. 2021A study on the plasticity of neural networkscited by 44
  7. 2024Weight Clipping for Deep Continual and Reinforcement Learningcited by 29
  8. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  9. 2024Exploring Quantization for Efficient Pre-Training of Transformer Language Modelscited by 10
  10. 2020Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizationscited by 9
  11. 2021Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  2. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  3. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 16
  4. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization
  2. 2025A Refined Analysis of Massive Activations in LLMs
  3. 2024Scaling Laws for Post Training Quantized Large Language Models
  4. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities
  5. 2026SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training
  6. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  8. 2025Achieving binary weight and activation for LLMs using Post-Training Quantization