mlsanityData to 7 Oct 2026

Differentiable Model Compression via Pseudo Quantization Noise

arXiv 2104.09987 ↗First posted 20 Apr 2021stat.ML cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

We propose DiffQ a differentiable method for model compression for quantizing model parameters without gradient approximations (e.g., Straight Through Estimator). We suggest adding independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. DiffQ is differentiable both with respect to the unquantized weights and the number of bits used. Given a single hyper-parameter balancing between the quantized model size and accuracy, DiffQ optimizes the number of bits used per individual weight or groups of weights, in end-to-end training. We experimentally verify that our method is competitive with STE based quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation. For instance, on the ImageNet dataset, DiffQ compresses a 12 layers transformer-based model by more than a factor of 8, (lower than 4 bits precision per weight on average), with a loss of 0.3% in model accuracy. Code is available at github.com/facebookresearch/diffq.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

49 of 72 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2016Wide Residual Networkscited by 3,057
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
37 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  6. 2016End-to-end Optimized Image Compressioncited by 673
  7. 2019Learned Step Size Quantizationcited by 604
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  11. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  12. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  13. 2018Model compression via distillation and quantizationcited by 287
  14. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  15. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  16. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  17. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  18. 2016Ternary Weight Networkscited by 195
  19. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  20. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
  21. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  22. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  23. 2019Variable Rate Deep Image Compression With a Conditional Autoencodercited by 109
  24. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  25. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  26. 2019Music Source Separation in the Waveform Domaincited by 98
  27. 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 87
  28. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  29. 2019And the Bit Goes Down: Revisiting the Quantization of Neural Networkscited by 78
  30. 2020HAWQV3: Dyadic Neural Network Quantizationcited by 59
  31. 2017WRPN: Wide Reduced-Precision Networkscited by 49
  32. 2020Differentiable Joint Pruning and Quantization for Hardware Efficiencycited by 45
  33. 2017Learning Discrete Weights Using the Local Reparameterization Trickcited by 29
  34. 2018NICE: Noise Injection and Clamping Estimation for Neural Network Quantizationcited by 24
  35. 2019Learning low-precision neural networks without Straight-Through Estimator(STE)cited by 15
  36. 2018UNIQ: Uniform Noise Injection for Non-Uniform Quantization of Neural Networkscited by 9
  37. 2019Improving singing voice separation using Deep U-Net and Wave-U-Net with data augmentationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 8; 2024: 9; 2025: 13; 2026: 3.

44 papers.

Sort:

By area:

  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  3. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  4. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  5. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  6. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  7. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformerscited by 14
  8. 2024Retraining-free Model Quantization via One-Shot Weight-Coupling Learningcited by 13
  9. 2022Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devicescited by 12
  10. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 11
  11. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Notcited by 10
  12. 2024HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAscited by 9
32 more
  1. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  2. 2024QGen: On the Ability to Generalize in Quantization Aware Trainingcited by 7
  3. 2023Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networkscited by 6
  4. 2024MetaAug: Meta-Data Augmentation for Post-Training Quantizationcited by 5
  5. 2022QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noisecited by 4
  6. 2022Neural Networks with Quantization Constraintscited by 4
  7. 2022Vertical Layering of Quantized Neural Networks for Heterogeneous Inferencecited by 4
  8. 2023QBitOpt: Fast and Accurate Bitwidth Reallocation during Trainingcited by 4
  9. 2025LOTION: Smoothing the Optimization Landscape for Quantized Trainingcited by 4
  10. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Trainingcited by 4
  11. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 4
  12. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsitycited by 3
  13. 2023Self-Compressing Neural Networkscited by 2
  14. 2025Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrixcited by 2
  15. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidthscited by 2
  16. 2023Rotation Invariant Quantization for Model Compressioncited by 1
  17. 2023On Calibration of Modern Quantized Efficient Neural Networkscited by 1
  18. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Trainingcited by 1
  19. 2024Scheduling Weight Transitions for Quantization-Aware Trainingcited by 1
  20. 2025Oscillations Make Neural Networks Robust to Quantizationcited by 1
  21. 2025A probabilistic framework for dynamic quantizationcited by 1
  22. 2025Merge-Friendly Post-Training Quantization for Multi-Target Domain Adaptationcited by 1
  23. 2025Starting Positions Matter: A Study on Better Weight Initialization for Neural Network Quantizationcited by 1
  24. 2022Error-aware Quantization through Noise Temperingcited by 0
  25. 2023Modular Quantization-Aware Training for 6D Object Pose Estimationcited by 0
  26. 2023RAND: Robustness Aware Norm Decay For Quantized Seq2seq Modelscited by 0
  27. 2023RepQ: Generalizing Quantization-Aware Training for Re-Parametrized Architecturescited by 0
  28. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMscited by 0
  29. 2025Semi-parametric Memory Consolidation: Towards Brain-like Deep Continual Learningcited by 0
  30. 2025Optimizing Deep Neural Networks using Safety-Guided Self Compressioncited by 0
  31. 2025Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Trainingcited by 0
  32. 2025MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Trainingcited by 0