# Training Dynamics Impact Post-Training Quantization Robustness

arXiv: https://arxiv.org/abs/2510.06213 · HTML: https://mlsanity.com/paper/2510.06213/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 76 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 30 of 53)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)

## Built on by (most cited first, 15 of 15)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting](https://mlsanity.com/paper/2605.02105/) (2026)
- [Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay](https://mlsanity.com/paper/2605.26097/) (2026)
- [(How) Learning Rates Regulate Catastrophic Overtraining](https://mlsanity.com/paper/2604.13627/) (2026)
- [Weight Decay Improves Language Model Plasticity](https://mlsanity.com/paper/2602.11137/) (2026)
- [Zero-Shot Quantization via Weight-Space Arithmetic](https://mlsanity.com/paper/2604.03420/) (2026)
- [Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin](https://mlsanity.com/paper/2606.09012/) (2026)
- [How Post-Training Shapes Biological Reasoning Models](https://mlsanity.com/paper/2606.16517/) (2026)
- [Why Does Post-Training Quantization Work?](https://mlsanity.com/paper/2609.11716/) (2026)
- [SandwichQuant: Which Parameters Matter Before and After Quantization?](https://mlsanity.com/paper/2608.24173/) (2026)
- [The Undetected Damage of Quantization on Retrieval and How to Fix It](https://mlsanity.com/paper/2609.24322/) (2026)
- [Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox](https://mlsanity.com/paper/2609.32060/) (2026)
- [Linger and Lose: Knowledge Collapse in Low-Bit Language Models](https://mlsanity.com/paper/2609.32902/) (2026)
- [Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization](https://mlsanity.com/paper/2610.05329/) (2026)
- [Detecting Unreliable Classifications from Quantized Models](https://mlsanity.com/paper/2610.18575/) (2026)

## Abstract (arXiv)

While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct a comprehensive analysis of quantization degradation across open-source language model training trajectories up to 32B parameters and 15T training tokens to accurately assess the relationship between training dynamics and quantization performance. Our key finding is that quantization errors in large-scale training runs are driven by a complex interplay between learning rate and other training hyperparameters. Specifically, once learning rates decay, validation loss and quantization error diverge, largely independent of training data scale. To investigate interventions on the training dynamics and identify specific configurations that can modulate quantization robustness favorably, we train our own models in controlled experiments up to 100B tokens. Our results challenge the assumption that increasing dataset scale inherently compromises quantization effectiveness, demonstrating instead that strategic training hyperparameter interventions can improve quantization quality at scale.