# CSQ: Growing Mixed-Precision Quantization Scheme with Bi-level Continuous Sparsification

arXiv: https://arxiv.org/abs/2212.02770 · HTML: https://mlsanity.com/paper/2212.02770/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019)
- [Zero-shot Adversarial Quantization](https://mlsanity.com/paper/2103.15263/) (2021)
- [BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization](https://mlsanity.com/paper/2102.10462/) (2021)
- [Growing Efficient Deep Networks by Structured Continuous Sparsification](https://mlsanity.com/paper/2007.15353/) (2020)
- [QUANOS- Adversarial Noise Sensitivity Driven Hybrid Quantization of Neural Networks](https://mlsanity.com/paper/2004.11233/) (2020)
- [Finding Non-Uniform Quantization Schemes using Multi-Task Gaussian Processes](https://mlsanity.com/paper/2007.07743/) (2020)

## Built on by (most cited first, 8 of 8)

- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detection](https://mlsanity.com/paper/2308.10515/) (2023)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [Block-Wise Mixed-Precision Quantization: Enabling High Efficiency for Practical ReRAM-based DNN Accelerators](https://mlsanity.com/paper/2310.12182/) (2023)
- [DQRM: Deep Quantized Recommendation Models](https://mlsanity.com/paper/2410.20046/) (2024)
- [MSQ: Memory-Efficient Bit Sparsification Quantization](https://mlsanity.com/paper/2507.22349/) (2025)
- [Fisher-aware Quantization for DETR Detectors with Critical-category Objectives](https://mlsanity.com/paper/2407.03442/) (2024)
- [TASQ: Temporal-Adaptive Bit Sparsification Quantization for Diffusion Models](https://mlsanity.com/paper/2608.03057/) (2026)

## Abstract (arXiv)

Mixed-precision quantization has been widely applied on deep neural networks (DNNs) as it leads to significantly better efficiency-accuracy tradeoffs compared to uniform quantization. Meanwhile, determining the exact precision of each layer remains challenging. Previous attempts on bit-level regularization and pruning-based dynamic precision adjustment during training suffer from noisy gradients and unstable convergence. In this work, we propose Continuous Sparsification Quantization (CSQ), a bit-level training method to search for mixed-precision quantization schemes with improved stability. CSQ stabilizes the bit-level mixed-precision training process with a bi-level gradual continuous sparsification on both the bit values of the quantized weights and the bit selection in determining the quantization precision of each layer. The continuous sparsification scheme enables fully-differentiable training without gradient approximation while achieving an exact quantized model in the end.A budget-aware regularization of total model size enables the dynamic growth and pruning of each layer's precision towards a mixed-precision quantization scheme of the desired size. Extensive experiments show CSQ achieves better efficiency-accuracy tradeoff than previous methods on multiple models and datasets.