# HiKonv: Maximizing the Throughput of Quantized Convolution With Novel Bit-wise Management and Computation

arXiv: https://arxiv.org/abs/2208.00763 · HTML: https://mlsanity.com/paper/2208.00763/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 22 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks](https://mlsanity.com/paper/1712.01507/) (2017)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edge](https://mlsanity.com/paper/1904.04421/) (2019)
- [EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions](https://mlsanity.com/paper/2005.02563/) (2020)
- [PULP-NN: Accelerating Quantized Neural Networks on Parallel Ultra-Low-Power RISC-V Processors](https://mlsanity.com/paper/1908.11263/) (2019)
- [Survey of Machine Learning Accelerators](https://mlsanity.com/paper/2009.00993/) (2020)
- [SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems](https://mlsanity.com/paper/1909.09709/) (2019)
- [VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization](https://mlsanity.com/paper/2005.08501/) (2020)
- [A Mixed-Precision RISC-V Processor for Extreme-Edge DNN Inference](https://mlsanity.com/paper/2010.04073/) (2020)
- [Loom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networks](https://mlsanity.com/paper/1706.07853/) (2017)
- [Face Recognition with Hybrid Efficient Convolution Algorithms on FPGAs](https://mlsanity.com/paper/1803.09004/) (2018)

## Abstract (arXiv)

Quantization for CNN has shown significant progress with the intention of reducing the cost of computation and storage with low-bitwidth data representations. There are, however, no systematic studies on how an existing full-bitwidth processing unit, such as ALU in CPUs and DSP in FPGAs, can be better utilized to deliver significantly higher computation throughput for convolution under various quantized bitwidths. In this study, we propose HiKonv, a unified solution that maximizes the throughput of convolution on a given underlying processing unit with low-bitwidth quantized data inputs through novel bit-wise management and parallel computation. We establish theoretical framework and performance models using a full-bitwidth multiplier for highly parallelized low-bitwidth convolution, and demonstrate new breakthroughs for high-performance computing in this critical domain. For example, a single 32-bit processing unit in CPU can deliver 128 binarized convolution operations (multiplications and additions) and 13 4-bit convolution operations with a single multiplication instruction, and a single 27x18 multiplier in the FPGA DSP can deliver 60, 8 or 2 convolution operations with 1, 4 or 8-bit inputs in one clock cycle. We demonstrate the effectiveness of HiKonv on both CPU and FPGA. On CPU, HiKonv outperforms the baseline implementation with 1 to 8-bit inputs and provides up to 7.6x and 1.4x performance improvements for 1-D convolution, and performs 2.74x and 3.19x over the baseline implementation for 4-bit signed and unsigned data inputs for 2-D convolution. On FPGA, HiKonv solution enables a single DSP to process multiple convolutions with a shorter processing latency. For binarized input, each DSP with HiKonv is equivalent up to 76.6 LUTs. Compared to the DAC-SDC 2020 champion model, HiKonv achieves a 2.37x throughput improvement and 2.61x DSP efficiency improvement, respectively.