mlsanityData to 7 Oct 2026

LSQ+: Improving low-bit quantization through learnable offsets and better initialization

arXiv 2004.09576 ↗First posted 20 Apr 2020cs.CV cs.LG stat.MLcited by 180

Not on the map. Why?

Abstract (arXiv)

Unlike ReLU, newer activation functions (like Swish, H-swish, Mish) that are frequently employed in popular efficient architectures can also result in negative activation values, with skewed positive and negative ranges. Typical learnable quantization schemes [PACT, LSQ] assume unsigned quantization for activations and quantize all negative activations to zero which leads to significant loss in performance. Naively using signed quantization to accommodate these negative values requires an extra sign bit which is expensive for low-bit (2-, 3-, 4-bit) quantization. To solve this problem, we propose LSQ+, a natural extension of LSQ, wherein we introduce a general asymmetric quantization scheme with trainable scale and offset parameters that can learn to accommodate the negative activations. Gradient-based learnable quantization schemes also commonly suffer from high instability or variance in the final training performance, hence requiring a great deal of hyper-parameter tuning to reach a satisfactory performance. LSQ+ alleviates this problem by using an MSE-based initialization scheme for the quantization parameters. We show that this initialization leads to significantly lower variance in final performance across multiple training runs. Overall, LSQ+ shows state-of-the-art results for EfficientNet and MixNet and also significantly outperforms LSQ for low-bit quantization of neural nets with Swish activations (e.g.: 1.8% gain with W4A4 quantization and upto 5.6% gain with W2A2 quantization of EfficientNet-B0 on ImageNet dataset). To the best of our knowledge, ours is the first work to quantize such architectures to extremely low bit-widths.

Key facts. Cited by 180 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 25; 2023: 32; 2024: 40; 2025: 42; 2026: 20.

180 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021A White Paper on Neural Network Quantizationcited by 453
  3. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  4. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  5. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  6. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  7. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  8. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  9. 2024GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splattingcited by 97
  10. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  11. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 91
  12. 2022FP8 Quantization: The Power of the Exponentcited by 87
38 more
  1. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  2. 2023Temporal Dynamic Quantization for Diffusion Modelscited by 81
  3. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  4. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  5. 2022Mixed-Precision Neural Network Quantization via Learned Layer-wise Importancecited by 53
  6. 2021Implicit Neural Video Compressioncited by 52
  7. 2023Pruning vs Quantization: Which is Better?cited by 45
  8. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 44
  9. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  10. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  11. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  12. 2025Quartet: Native FP4 Training Can Be Optimal for Large Language Modelscited by 37
  13. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  14. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  15. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 32
  16. 2023FP8 versus INT8 for efficient deep learning inferencecited by 31
  17. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  18. 2024Boosting Neural Representations for Videos with a Conditional Decodercited by 29
  19. 2021Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formatscited by 26
  20. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 26
  21. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  22. 2023A Practical Mixed Precision Algorithm for Post-Training Quantizationcited by 23
  23. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23
  24. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23
  25. 2022Make RepVGG Greater Again: A Quantization-aware Approachcited by 21
  26. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
  27. 2023PowerQuant: Automorphism Search for Non-Uniform Quantizationcited by 20
  28. 2021CPT: Efficient Deep Neural Network Training via Cyclic Precisioncited by 19
  29. 2021BatchQuant: Quantized-for-all Architecture Search with Robust Quantizercited by 19
  30. 2021Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolutioncited by 19
  31. 2021Distance-aware Quantizationcited by 19
  32. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  33. 2021Mesa: A Memory-saving Training Framework for Transformerscited by 16
  34. 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
  35. 2021ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNscited by 15
  36. 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
  37. 2023Quantized Feature Distillation for Network Quantizationcited by 13
  38. 2024QT-DoG: Quantization-aware Training for Domain Generalizationcited by 13

50 most cited of 180 All 180 (JSON)