mlsanityData to 14 Oct 2026

Quantization-Aware Pre-Training with Constrained Empirical Weight Distribution

arXiv 2609.32659 ↗First posted 26 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Quantization-Aware Pre-Training (QAPT) can increase the inference efficiency of DNNs, but a problematic behaviour known as rounding boundary weight oscillation can introduce detrimental noise into the training process and significantly reduce convergence speed. While existing methods can reduce this detrimental noise, they either introduce additional hyperparameters or memory overhead, or cannot consistently improve model accuracy. In this work, we propose optimization with Constrained Empirical Weight disTribution (CEWT), the first hyperparameter-free memory-overhead-free oscillation suppression method that consistently improves QAPT performance: an optimizer post-update step that projects weights to the nearest point in weight space whose empirical distribution (histogram) matches a zero-mean Gaussian. Our key insight is many quantizers are designed with the implicit assumption that the to-be-quantized data are permutations of samples from a zero-mean Gaussian, and this assumption is not true during QAPT. By enforcing the zero-mean Gaussian prior as a hard constraint, CEWT can suppress this detrimental noise. Empirical results on various combinations of SOTA quantizers and hypersphere optimizers suggest, that with a geomean increase of 4% in training time, CEWT can consistently reduce the pre-training perplexity (by an average of 2.5 and up to 21 points) of low-precision (down to 1-bit activations and weights and up to 610M parameters) LLaMA/GPT models without introducing any hyperparameters or storage overhead. Code is available at https://github.com/1733116199/cewt

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

34 of 46 references.

References · 34

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2019Learned Step Size Quantizationcited by 628
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  10. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
22 more
  1. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  2. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  3. 2023A Spectral Condition for Feature Learningcited by 109
  4. 2024Scaling Laws for Precisioncited by 95
  5. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  6. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  7. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  8. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 61
  9. 2023Training Transformers with 4-bit Integerscited by 60
  10. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  11. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  12. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 42
  13. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 39
  14. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  15. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  16. 2026Controlled LLM Training on Spectral Spherecited by 31
  17. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 12
  18. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  19. 2026WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Pointscited by 3
  20. 2026JacQuant: STE-Free Quantization-Aware Training via Learned Jacobian Surrogatescited by 2
  21. 2025Oscillations Make Neural Networks Robust to Quantizationcited by 1
  22. 2026Boosting Entropy with Bell Box Quantizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  2. 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedom
  3. 2025Compute-Optimal Quantization-Aware Training
  4. 2021Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
  5. 2026Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization
  6. 2024Gradient-Free Training of Quantized Neural Networks
  7. 2026Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
  8. 2024MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization