# BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization

arXiv: https://arxiv.org/abs/2002.03090 · HTML: https://mlsanity.com/paper/2002.03090/

## Key facts

- Cited by 14 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 8 Feb 2020.

## Built on by (most cited first, 14 of 14)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [SDQ: Stochastic Differentiable Quantization with Mixed Precision](https://mlsanity.com/paper/2206.04459/) (2022)
- [A Silicon Photonic Accelerator for Convolutional Neural Networks with Heterogeneous Quantization](https://mlsanity.com/paper/2205.11244/) (2022)
- [A Low Memory Footprint Quantized Neural Network for Depth Completion of Very Sparse Time-of-Flight Depth Maps](https://mlsanity.com/paper/2205.12918/) (2022)
- [Efficient and Effective Methods for Mixed Precision Neural Network Quantization for Faster, Energy-efficient Inference](https://mlsanity.com/paper/2301.13330/) (2023)
- [Free Bits: Latency Optimization of Mixed-Precision Quantized Neural Networks on the Edge](https://mlsanity.com/paper/2307.02894/) (2023)
- [Unsupervised ANN-Based Equalizer and Its Trainable FPGA Implementation](https://mlsanity.com/paper/2304.06987/) (2023)
- [Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment](https://mlsanity.com/paper/2606.19558/) (2026)
- [DNN Quantization with Attention](https://mlsanity.com/paper/2103.13322/) (2021)
- [AdaQAT: Adaptive Bit-Width Quantization-Aware Training](https://mlsanity.com/paper/2404.16876/) (2024)
- [CNN-Based Equalization for Communications: Achieving Gigabit Throughput with a Flexible FPGA Hardware Architecture](https://mlsanity.com/paper/2405.02323/) (2024)
- [APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inference](https://mlsanity.com/paper/2201.08830/) (2022)
- [Schr"odinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training](https://mlsanity.com/paper/2204.13666/) (2022)
- [FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs](https://mlsanity.com/paper/2211.06982/) (2022)

## Abstract (arXiv)

Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy. Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint. We demonstrate that our method learns thrifty representations while maintaining accuracy. With ImageNet, the method produces an average per layer bitlength of 4.13, 3.76 and 4.36 bits on AlexNet, ResNet18 and MobileNet V2 respectively, remaining within 2.0%, 0.5% and 0.5% of the base TOP-1 accuracy.