mlsanityData to 14 Oct 2026

MSQ: Memory-Efficient Bit Sparsification Quantization

arXiv 2507.22349 ↗First posted 30 Jul 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

As deep neural networks (DNNs) see increased deployment on mobile and edge devices, optimizing model efficiency has become crucial. Mixed-precision quantization is widely favored, as it offers a superior balance between efficiency and accuracy compared to uniform quantization. However, finding the optimal precision for each layer is challenging. Recent studies utilizing bit-level sparsity have shown promise, yet they often introduce substantial training complexity and high GPU memory requirements. In this paper, we propose Memory-Efficient Bit Sparsification Quantization (MSQ), a novel approach that addresses these limitations. MSQ applies a round-clamp quantizer to enable differentiable computation of the least significant bits (LSBs) from model weights. It further employs regularization to induce sparsity in these LSBs, enabling effective precision reduction without explicit bit-level parameter splitting. Additionally, MSQ incorporates Hessian information, allowing the simultaneous pruning of multiple LSBs to further enhance training efficiency. Experimental results show that MSQ achieves up to 8.00x reduction in trainable parameters and up to 86% reduction in training time compared to previous bit-level quantization, while maintaining competitive accuracy and compression rates. This makes it a practical solution for training efficient DNNs on resource-constrained devices.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

26 of 32 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  4. 2017Squeeze-and-Excitation Networkscited by 4,789
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2019Searching for MobileNetV3cited by 1,994
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2019Learned Step Size Quantizationcited by 628
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  11. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  12. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
14 more
  1. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  2. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  3. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  4. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  5. 2019Quantization Networkscited by 153
  6. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  7. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  8. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 61
  9. 2020DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUscited by 58
  10. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  11. 2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 55
  12. 2021BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantizationcited by 39
  13. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  14. 2022CSQ: Growing Mixed-Precision Quantization Scheme with Bi-level Continuous Sparsificationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgetscited by 1
  2. 2026TASQ: Temporal-Adaptive Bit Sparsification Quantization for Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Post-Training Sparsity-Aware Quantization
  2. 2021BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch
  3. 2025InfoQ: Mixed-Precision Quantization via Global Information Flow
  4. 2021A White Paper on Neural Network Quantization
  5. 2025Low-bit Model Quantization for Deep Neural Networks: A Survey
  6. 2025Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference
  7. 2022SDQ: Stochastic Differentiable Quantization with Mixed Precision
  8. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmark