mlsanityData to 7 Oct 2026

High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator

arXiv 2510.10693 ↗First posted 12 Oct 2025stat.ML cond-mat.dis-nn cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

54 of 71 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
38 more
  1. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  2. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  3. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  4. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  5. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  6. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  9. 2017Training Quantized Nets: A Deeper Understandingcited by 74
  10. 2022Universality of empirical risk minimizationcited by 73
  11. 2021Universal Approximation of Functions on Setscited by 62
  12. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  13. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  14. 2020Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimatorcited by 37
  15. 2025Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantizationcited by 36
  16. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  17. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  18. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  19. 2019Relation Embedding with Dihedral Group in Knowledge Graphcited by 25
  20. 2022Learning strides in convolutional neural networkscited by 25
  21. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  22. 2022Enhance the Visual Representation via Discrete Adversarial Trainingcited by 23
  23. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
  24. 2018Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networkscited by 20
  25. 2019Mirror Descent View for Neural Network Quantizationcited by 15
  26. 2018A Solvable High-Dimensional Model of GANcited by 13
  27. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13
  28. 2023Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealingcited by 12
  29. 2025Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysiscited by 12
  30. 2017Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCAcited by 11
  31. 2018On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networkscited by 11
  32. 2021Demystifying and Generalizing BinaryConnectcited by 11
  33. 2019Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimizationcited by 9
  34. 2020Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classificationcited by 8
  35. 2019Straight-Through Estimator as Projected Wasserstein Gradient Flowcited by 7
  36. 2022Gradient Estimation for Binary Latent Variables via Gradient Variance Clippingcited by 7
  37. 2024QGen: On the Ability to Generalize in Quantization Aware Trainingcited by 7
  38. 2018Subspace Estimation from Incomplete Observations: A High-Dimensional Analysiscited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantizationcited by 3
  2. 2026Improving Generalization by Permutation Routing Across Model Copiescited by 0
  3. 2026Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuningcited by 0
  4. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0