# AutoQ: Automated Kernel-Wise Neural Network Quantization

arXiv: https://arxiv.org/abs/1902.05690 · HTML: https://mlsanity.com/paper/1902.05690/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2019.

## Built on by (most cited first, 30 of 63)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization](https://mlsanity.com/paper/2110.06554/) (2021)
- [Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance](https://mlsanity.com/paper/2203.08368/) (2022)
- [FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization](https://mlsanity.com/paper/2306.00317/) (2023)
- [Q-ViT: Fully Differentiable Quantization for Vision Transformer](https://mlsanity.com/paper/2201.07703/) (2022)
- [CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution](https://mlsanity.com/paper/2207.10345/) (2022)
- [EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization](https://mlsanity.com/paper/2307.10554/) (2023)
- [Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learning](https://mlsanity.com/paper/2302.04453/) (2023)
- [Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision](https://mlsanity.com/paper/2307.00331/) (2023)
- [Patch-wise Mixed-Precision Quantization of Vision Transformer](https://mlsanity.com/paper/2305.06559/) (2023)
- [Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss](https://mlsanity.com/paper/2109.02100/) (2021)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices](https://mlsanity.com/paper/2311.18129/) (2023)
- [Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks](https://mlsanity.com/paper/2112.15139/) (2021)
- [HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAs](https://mlsanity.com/paper/2405.00645/) (2024)
- [Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivity](https://mlsanity.com/paper/2501.06357/) (2025)
- [Fast Jet Tagging with MLP-Mixers on FPGAs](https://mlsanity.com/paper/2503.03103/) (2025)
- [LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagation](https://mlsanity.com/paper/2401.11243/) (2024)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Mixed-Precision Quantization for Deep Vision Models with Integer Quadratic Programming](https://mlsanity.com/paper/2307.05657/) (2023)
- [TransAxx: Efficient Transformers with Approximate Computing](https://mlsanity.com/paper/2402.07545/) (2024)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Single-path Bit Sharing for Automatic Loss-aware Model Compression](https://mlsanity.com/paper/2101.04935/) (2021)
- [Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization](https://mlsanity.com/paper/2103.02904/) (2021)
- [RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions](https://mlsanity.com/paper/2111.00153/) (2021)
- [Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks](https://mlsanity.com/paper/2407.01054/) (2024)
- [Sub-microsecond Transformers for Jet Tagging on FPGAs](https://mlsanity.com/paper/2510.24784/) (2025)
- [Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition](https://mlsanity.com/paper/2206.15408/) (2022)
- [Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks](https://mlsanity.com/paper/2307.13337/) (2023)
- [M4BRAM: Mixed-Precision Matrix-Matrix Multiplication in FPGA Block RAMs](https://mlsanity.com/paper/2311.02758/) (2023)

## Abstract (arXiv)

Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.