mlsanityData to 14 Oct 2026

Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training

arXiv 2206.06501 ↗First posted 13 Jun 2022cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Data clipping is crucial in reducing noise in quantization operations and improving the achievable accuracy of quantization-aware training (QAT). Current practices rely on heuristics to set clipping threshold scalars and cannot be shown to be optimal. We propose Optimally Clipped Tensors And Vectors (OCTAV), a recursive algorithm to determine MSE-optimal clipping scalars. Derived from the fast Newton-Raphson method, OCTAV finds optimal clipping scalars on the fly, for every tensor, at every iteration of the QAT routine. Thus, the QAT algorithm is formulated with provably minimum quantization noise at each step. In addition, we reveal limitations in common gradient estimation techniques in QAT and propose magnitude-aware differentiation as a remedy to further improve accuracy. Experimentally, OCTAV-enabled QAT achieves state-of-the-art accuracy on multiple tasks. These include training-from-scratch and retraining ResNets and MobileNets on ImageNet, and Squad fine-tuning using BERT models, where OCTAV-enabled QAT consistently preserves accuracy at low precision (4-to-6-bits). Our results require no modifications to the baseline training recipe, except for the insertion of quantization operations where appropriate.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

27 of 46 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2019Searching for MobileNetV3cited by 1,994
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2019Learned Step Size Quantizationcited by 628
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  10. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  11. 2015Deep Learning with Limited Numerical Precisioncited by 376
  12. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
15 more
  1. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  2. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  3. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  4. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  5. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 164
  6. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 80
  7. 2018Benchmark Analysis of Representative Deep Neural Network Architecturescited by 77
  8. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  9. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  10. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 44
  11. 2021Pareto-Optimal Quantized ResNet Is Mostly 4-bitcited by 26
  12. 2018Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithmcited by 15
  13. 2019Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networkscited by 11
  14. 2020DBQ: A Differentiable Branch Quantizer for Lightweight Deep Neural Networkscited by 8
  15. 2019AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inferencecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 13; 2025: 4; 2026: 4.

26 papers.

Sort:

By area:

  1. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  2. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  3. 2023Similarity search in the blink of an eye with compressed indicescited by 21
  4. 2023PowerQuant: Automorphism Search for Non-Uniform Quantizationcited by 20
  5. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  6. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  7. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  8. 2024eXmY: A Data Type and Technique for Arbitrary Bit Precision Quantizationcited by 7
  9. 2025LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inferencecited by 7
  10. 2024Custom Gradient Estimators are Straight-Through Estimators in Disguisecited by 6
  11. 2025LOTION: Smoothing the Optimization Landscape for Quantized Trainingcited by 6
  12. 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedomcited by 3
14 more
  1. 2024On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 3
  2. 2024One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Trainingcited by 2
  3. 2024RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantizationcited by 2
  4. 2024Communication-Efficient Federated Learning via Clipped Uniform Quantizationcited by 2
  5. 2024Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learningcited by 2
  6. 2024Continuous Approximations for Improving Quantization Aware Training of LLMscited by 2
  7. 2023INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformerscited by 1
  8. 2024On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performancecited by 1
  9. 2025DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Schedulingcited by 1
  10. 2024Predicting High-precision Depth on Low-Precision Devices Using 2D Hilbert Curvescited by 0
  11. 2026Why ReLU? A Bit-Model Dichotomy for Deep Network Trainingcited by 0
  12. 2026StatQAT: Statistical Quantizer Optimization for Deep Networkscited by 0
  13. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0
  14. 2026IMC-CLINIC: Coupled Loss-Informed Newton Iterations for Clipping in Analog In-Memory Computingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection
  2. 2025Compute-Optimal Quantization-Aware Training
  3. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
  4. 2026Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
  5. 2026Zero-Shot Quantization via Weight-Space Arithmetic
  6. 2025QuantKAN: A Unified Quantization Framework for Kolmogorov Arnold Networks
  7. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  8. 2026SCULPT: Training Edge Vision Models for Post-Training Quantization Readiness