# Learnable Companding Quantization for Accurate Low-bit Neural Networks

arXiv: https://arxiv.org/abs/2103.07156 · HTML: https://mlsanity.com/paper/2103.07156/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Towards Effective Low-bitwidth Convolutional Neural Networks](https://mlsanity.com/paper/1711.00205/) (2017)
- [Training Quantized Neural Networks with a Full-precision Auxiliary Module](https://mlsanity.com/paper/1903.11236/) (2019)
- [Deep Neural Network Compression with Single and Multiple Level Quantization](https://mlsanity.com/paper/1803.03289/) (2018)
- [Learning low-precision neural networks without Straight-Through Estimator(STE)](https://mlsanity.com/paper/1903.01061/) (2019)

## Built on by (most cited first, 30 of 42)

- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [Fast Matrix Multiplications for Lookup Table-Quantized LLMs](https://mlsanity.com/paper/2407.10960/) (2024)
- [SDQ: Stochastic Differentiable Quantization with Mixed Precision](https://mlsanity.com/paper/2206.04459/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Knowledge Condensation Distillation](https://mlsanity.com/paper/2207.05409/) (2022)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [Quantized Feature Distillation for Network Quantization](https://mlsanity.com/paper/2307.10638/) (2023)
- [CA-SpaceNet: Counterfactual Analysis for 6D Pose Estimation in Space](https://mlsanity.com/paper/2207.07869/) (2022)
- [QT-DoG: Quantization-aware Training for Domain Generalization](https://mlsanity.com/paper/2410.06020/) (2024)
- [Q-LIC: Quantizing Learned Image Compression with Channel Splitting](https://mlsanity.com/paper/2205.14510/) (2022)
- [DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables](https://mlsanity.com/paper/2304.09049/) (2023)
- [GenQ: Quantization in Low Data Regimes with Generative Synthetic Data](https://mlsanity.com/paper/2312.05272/) (2023)
- [SQuAT: Sharpness- and Quantization-Aware Training for BERT](https://mlsanity.com/paper/2210.07171/) (2022)
- [MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization](https://mlsanity.com/paper/2311.06798/) (2023)
- [Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies](https://mlsanity.com/paper/2209.01404/) (2022)
- [Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart](https://mlsanity.com/paper/2412.15846/) (2024)
- [Task Vector Quantization for Memory-Efficient Model Merging](https://mlsanity.com/paper/2503.06921/) (2025)
- [LG-LSQ: Learned Gradient Linear Symmetric Quantization](https://mlsanity.com/paper/2202.09009/) (2022)
- [BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation](https://mlsanity.com/paper/2207.01394/) (2022)
- [Model Compression Methods for YOLOv5: A Review](https://mlsanity.com/paper/2307.11904/) (2023)
- [Boolean Variation and Boolean Logic BackPropagation](https://mlsanity.com/paper/2311.07427/) (2023)
- [HEQuant: Marrying Homomorphic Encryption and Quantization for Communication-Efficient Private Inference](https://mlsanity.com/paper/2401.15970/) (2024)
- [Differentiable Search for Finding Optimal Quantization Strategy](https://mlsanity.com/paper/2404.08010/) (2024)
- [Scheduling Weight Transitions for Quantization-Aware Training](https://mlsanity.com/paper/2404.19248/) (2024)
- [GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networks](https://mlsanity.com/paper/2508.14004/) (2025)
- [Deep Lookup Network](https://mlsanity.com/paper/2509.13662/) (2025)
- [Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks](https://mlsanity.com/paper/2208.07265/) (2022)
- [Weight Fixing Networks](https://mlsanity.com/paper/2210.13554/) (2022)
- [Probabilistic Weight Fixing: Large-scale training of neural network weight uncertainties for quantization](https://mlsanity.com/paper/2309.13575/) (2023)
- [Coded Deep Learning: Framework and Algorithm](https://mlsanity.com/paper/2501.09849/) (2025)

## Abstract (arXiv)

Quantizing deep neural networks is an effective method for reducing memory consumption and improving inference speed, and is thus useful for implementation in resource-constrained devices. However, it is still hard for extremely low-bit models to achieve accuracy comparable with that of full-precision models. To address this issue, we propose learnable companding quantization (LCQ) as a novel non-uniform quantization method for 2-, 3-, and 4-bit models. LCQ jointly optimizes model weights and learnable companding functions that can flexibly and non-uniformly control the quantization levels of weights and activations. We also present a new weight normalization technique that allows more stable training for quantization. Experimental results show that LCQ outperforms conventional state-of-the-art methods and narrows the gap between quantized and full-precision models for image classification and object detection tasks. Notably, the 2-bit ResNet-50 model on ImageNet achieves top-1 accuracy of 75.1% and reduces the gap to 1.7%, allowing LCQ to further exploit the potential of non-uniform quantization.