mlsanityData to 14 Oct 2026

Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks

arXiv 1908.05033 ↗First posted 14 Aug 2019cs.CV cs.LG eess.IVcited by 238

Not on the map. Why?

Abstract (arXiv)

Hardware-friendly network quantization (e.g., binary/uniform quantization) can efficiently accelerate the inference and meanwhile reduce memory consumption of the deep neural networks, which is crucial for model deployment on resource-limited devices like mobile phones. However, due to the discreteness of low-bit quantization, existing quantization methods often face the unstable training process and severe performance degradation. To address this problem, in this paper we propose Differentiable Soft Quantization (DSQ) to bridge the gap between the full-precision and low-bit networks. DSQ can automatically evolve during training to gradually approximate the standard quantization. Owing to its differentiable property, DSQ can help pursue the accurate gradients in backward propagation, and reduce the quantization loss in forward process with an appropriate clipping range. Extensive experiments over several popular network structures show that training low-bit neural networks with DSQ can consistently outperform state-of-the-art quantization methods. Besides, our first efficient implementation for deploying 2 to 4-bit DSQ on devices with ARM architecture achieves up to 1.7× speed up, compared with the open-source 8-bit high-performance inference framework NCNN. [31]

Key facts. Cited by 238 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 45; 2022: 45; 2023: 58; 2024: 44; 2025: 30; 2026: 16.

238 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  3. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  4. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  5. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  6. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  7. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 170
  8. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  9. 2021Invertible Image Signal Processingcited by 94
  10. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  11. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  12. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Modelscited by 78
38 more
  1. 2023TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Modelscited by 69
  2. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  3. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  4. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  5. 2023Training Transformers with 4-bit Integerscited by 60
  6. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  7. 2021AdvDrop: Adversarial Attack to DNNs by Dropping Informationcited by 56
  8. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  9. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  10. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 52
  11. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  12. 2023LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Versioncited by 50
  13. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 49
  14. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  15. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  16. 2021A comprehensive review of Binary Neural Networkcited by 45
  17. 2021IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantizationcited by 43
  18. 2021Learnable Companding Quantization for Accurate Low-bit Neural Networkscited by 42
  19. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  20. 2021OMPQ: Orthogonal Mixed Precision Quantizationcited by 39
  21. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  22. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  23. 2023Ternary Spike: Learning Ternary Spikes for Spiking Neural Networkscited by 38
  24. 2024Convolutional Differentiable Logic Gate Networkscited by 37
  25. 2022Towards Semantic Communications: Deep Learning-Based Image Semantic Codingcited by 36
  26. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  27. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  28. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  29. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 28
  30. 2023RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networkscited by 28
  31. 2021Distribution-sensitive Information Retention for Accurate Binary Neural Networkcited by 25
  32. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  33. 2024EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Modelscited by 23
  34. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  35. 2021Bit-Mixer: Mixed-precision networks with runtime bit-width selectioncited by 22
  36. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  37. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 22
  38. 2025GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuningcited by 21

50 most cited of 238 All 238 (JSON)