mlsanityData to 14 Oct 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

arXiv 2610.05329 ↗First posted 4 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide a simple way to improve the pretraining-to-quantization transition. But the mechanism behind weight averaging remains insufficiently explained. This leads to inconsistent and fragile performance gains, thereby preventing practitioners from applying such a technique confidently. As a response, we formulate weight averaging as a trade-off between retaining training progress and improving robustness under perturbation. We further derive a continuous family of averaging kernels that unifies conventional strategies and achieves the Pareto frontier between the two competing goals. Critically, a theoretical framework for performing weight averaging under PTQ is developed. It can be shown that coarser quantization is more susceptible to perturbations, whereas finer quantization could be less affected. Thus, our results could provide unified theoretical guidance for performing weight averaging under different PTQ conditions. Experiments validate both the predicted behavior and the proposed averaging strategy. Code is available at https://github.com/MOFA-LAB/weight-averaging-for-ptq.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 56 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

44 of 56 references.

References · 44

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  12. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
32 more
  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  5. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  6. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  7. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  8. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  9. 2024Scaling Laws for Precisioncited by 95
  10. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  11. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 60
  12. 2023Intriguing Properties of Quantization at Scalecited by 49
  13. 2018Post-training 4-bit quantization of convolution networks for rapid-deploymentcited by 42
  14. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 41
  15. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  16. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  17. 2025The Coverage Principle: How Pre-Training Enables Post-Trainingcited by 30
  18. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  19. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 22
  20. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  21. 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 17
  22. 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantizationcited by 17
  23. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15
  24. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  25. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  26. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  27. 2025When, Where and Why to Average Weights?cited by 8
  28. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 8
  29. 2023Quantized Low-Rank Multivariate Regression with Random Ditheringcited by 5
  30. 2025Learning Dynamics in Continual Pre-Training for Large Language Modelscited by 5
  31. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 3
  32. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
  2. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs
  3. 2025Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach
  4. 2024Scaling Laws for Post Training Quantized Large Language Models
  5. 2024Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners
  6. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices
  7. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
  8. 2024Evaluating Quantized Large Language Models