mlsanityData to 7 Oct 2026

Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation

arXiv 2111.14826 ↗First posted 29 Nov 2021cs.CV cs.AI cs.LGcited by 78

Not on the map. Why?

Abstract (arXiv)

The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonuniform quantization methods overlook the complicated projection process in implementing the nonuniformly quantized weights/activations, which incurs non-negligible time and space overhead in hardware deployment. In this study, we propose Nonuniform-to-Uniform Quantization (N2UQ), a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient as the uniform quantization for model inference. We achieve this through learning the flexible in-equidistant input thresholds to better fit the underlying distribution while quantizing these real-valued inputs into equidistant output levels. To train the quantized network with learnable input thresholds, we introduce a generalized straight-through estimator (G-STE) for intractable backward derivative calculation w.r.t. threshold parameters. Additionally, we consider entropy preserving regularization to further reduce information loss in weight quantization. Even under this adverse constraint of imposing uniformly quantized weights and activations, our N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.5~1.7 on ImageNet, demonstrating the contribution of N2UQ design. Code and models are available at: https://github.com/liuzechun/Nonuniform-to-Uniform-Quantization.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

49 of 58 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
37 more
  1. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  6. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  11. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  13. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  14. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  15. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  16. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  17. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
  18. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  19. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  20. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  21. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  22. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  23. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 97
  24. 2019XNOR-Net++: Improved Binary Neural Networkscited by 94
  25. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  26. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  27. 2018MEAL: Multi-Model Ensemble via Adversarial Learningcited by 52
  28. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  29. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  30. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 40
  31. 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 38
  32. 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
  33. 2020MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?cited by 32
  34. 2021Distance-aware Quantizationcited by 19
  35. 2020BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNscited by 17
  36. 2021Design and Operation of Shared Machine Learning Clusters on Campuscited by 12
  37. 2019An efficient floating point multiplier design for high speed applications using Karatsuba algorithm and Urdhva-Tiryagbhyam algorithmcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 20; 2024: 27; 2025: 21; 2026: 6.

78 papers.

Sort:

By area:

  1. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  2. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  3. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  4. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  5. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  6. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  7. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  8. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  9. 2024SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spikingcited by 29
  10. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  11. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23
  12. 2023A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platformscited by 21
38 more
  1. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  2. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  3. 2024ELMGS: Enhancing memory and computation scaLability through coMpression for 3D Gaussian Splattingcited by 19
  4. 2023IRGen: Generative Modeling for Image Retrievalcited by 18
  5. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  6. 2025PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancementcited by 17
  7. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  8. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  9. 2024Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptationcited by 15
  10. 2023Quantized Feature Distillation for Network Quantizationcited by 13
  11. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  12. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  13. 2024PassionSR: Post-Training Quantization with Adaptive Scale in One-Step Diffusion based Image Super-Resolutioncited by 8
  14. 2023MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantizationcited by 7
  15. 2023SEAM: Searching Transferable Mixed-Precision Quantization Policy through Large Margin Regularizationcited by 6
  16. 2023Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networkscited by 6
  17. 2024Custom Gradient Estimators are Straight-Through Estimators in Disguisecited by 6
  18. 2024SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantizationcited by 6
  19. 2024Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpartcited by 6
  20. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
  21. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 6
  22. 2023GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samplescited by 4
  23. 2023Low-Precision Mixed-Computation Models for Inference on Edgecited by 4
  24. 2024decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Pointscited by 4
  25. 2025Q-MambaIR: Accurate Quantized Mamba for Efficient Image Restorationcited by 4
  26. 2026Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformerscited by 4
  27. 2023Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inferencecited by 3
  28. 2024CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inferencecited by 3
  29. 2025Rotate, Clip, and Partition: Towards W2A4KV4 Quantization by Integrating Rotation and Learnable Non-uniform Quantizercited by 3
  30. 2025TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precisioncited by 3
  31. 2025MoQE: Improve Quantization Model performance via Mixture of Quantization Expertscited by 3
  32. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  33. 2023Efficient and Robust Quantization-aware Training via Adaptive Coreset Selectioncited by 2
  34. 2024HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inferencecited by 2
  35. 2024SQUAT: Stateful Quantization-Aware Training in Recurrent Spiking Neural Networkscited by 2
  36. 2024Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Othercited by 2
  37. 2024Towards Accurate and Efficient Sub-8-Bit Integer Trainingcited by 2
  38. 2025Extending Straight-Through Estimation for Robust Neural Networks on Analog CIM Hardwarecited by 2

50 most cited of 78 All 78 (JSON)