mlsanityData to 7 Oct 2026

BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization

arXiv 2102.10462 ↗First posted 20 Feb 2021cs.LG cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

Mixed-precision quantization can potentially achieve the optimal tradeoff between performance and compression rate of deep neural networks, and thus, have been widely investigated. However, it lacks a systematic method to determine the exact quantization scheme. Previous methods either examine only a small manually-designed search space or utilize a cumbersome neural architecture search to explore the vast search space. These approaches cannot lead to an optimal quantization scheme efficiently. This work proposes bit-level sparsity quantization (BSQ) to tackle the mixed-precision quantization from a new angle of inducing bit-level sparsity. We consider each bit of quantized weights as an independent trainable variable and introduce a differentiable bit-sparsity regularizer. BSQ can induce all-zero bits across a group of weight elements and realize the dynamic precision reduction, leading to a mixed-precision quantization scheme of the original model. Our method enables the exploration of the full mixed-precision space with a single gradient-based optimization process, with only one hyperparameter to tradeoff the performance and compression. BSQ achieves both higher accuracy and higher bit reduction on various model architectures on the CIFAR-10 and ImageNet datasets comparing to previous methods.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 27 of its 31 references that are in the map. First posted on arXiv 20 Feb 2021.

Builds on

27 of 31 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  9. 2019Learned Step Size Quantizationcited by 604
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  11. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  12. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
15 more
  1. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  2. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  3. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  4. 2018Model compression via distillation and quantizationcited by 287
  5. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  6. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  7. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  8. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  9. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 95
  10. 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
  11. 2018Value-aware Quantization for Training and Inference of Neural Networkscited by 49
  12. 2017Coordinating Filters for Faster Deep Neural Networkscited by 24
  13. 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
  14. 2018Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithmcited by 15
  15. 2016Effective Quantization Methods for Recurrent Neural Networkscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 8; 2024: 1; 2025: 7; 2026: 7.

37 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  3. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  4. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 88
  5. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  6. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  7. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  8. 2022SDQ: Stochastic Differentiable Quantization with Mixed Precisioncited by 27
  9. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  10. 2021Neural Network Quantization for Efficient Inference: A Surveycited by 13
  11. 2021Multi-Objective Optimization of ReRAM Crossbars for Robust DNN Inferencing under Stochastic Noisecited by 12
  12. 2022Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teachingcited by 12
25 more
  1. 2023Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devicescited by 11
  2. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  3. 2021BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratchcited by 10
  4. 2021HERO: Hessian-Enhanced Robust Optimization for Unifying and Improving Generalization and Quantization Performancecited by 8
  5. 2022CSQ: Growing Mixed-Precision Quantization Scheme with Bi-level Continuous Sparsificationcited by 7
  6. 2025LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bitscited by 6
  7. 2025Mixed precision accumulation for neural network inference guided by componentwise forward error analysiscited by 6
  8. 2023QBitOpt: Fast and Accurate Bitwidth Reallocation during Trainingcited by 4
  9. 2026STQuant: Spatio-Temporal Adaptive Framework for Optimizer Quantization in Large Multimodal Model Trainingcited by 4
  10. 2022AMED: Automatic Mixed-Precision Quantization for Edge Devicescited by 3
  11. 2023Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantizationcited by 2
  12. 2025MSQ: Memory-Efficient Bit Sparsification Quantizationcited by 2
  13. 2026dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formatscited by 2
  14. 2021A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rankcited by 1
  15. 2024Fisher-aware Quantization for DETR Detectors with Critical-category Objectivescited by 1
  16. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  17. 2026GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgetscited by 1
  18. 2022Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Trainingcited by 0
  19. 2023HCE: Improving Performance and Efficiency with Heterogeneously Compressed Neural Network Ensemblecited by 0
  20. 2023AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networkscited by 0
  21. 2023SONIQ: System-Optimized Noise-Injected Ultra-Low-Precision Quantization with Full-Precision Paritycited by 0
  22. 2025Cauchy-Schwarz Regularizerscited by 0
  23. 2026UFO: Unlocking Ultra-Efficient Quantized Private Inference with Protocol and Algorithm Co-Optimizationcited by 0
  24. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMscited by 0
  25. 2026TASQ: Temporal-Adaptive Bit Sparsification Quantization for Diffusion Modelscited by 0