# Free Bits: Latency Optimization of Mixed-Precision Quantized Neural Networks on the Edge

arXiv: https://arxiv.org/abs/2307.02894 · HTML: https://mlsanity.com/paper/2307.02894/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks](https://mlsanity.com/paper/1903.08066/) (2019)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [Rethinking Differentiable Search for Mixed-Precision Neural Networks](https://mlsanity.com/paper/2004.05795/) (2020)
- [DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs](https://mlsanity.com/paper/2008.07127/) (2020)
- [Ternary Neural Networks for Resource-Efficient AI Applications](https://mlsanity.com/paper/1609.00222/) (2016)
- [Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers](https://mlsanity.com/paper/1905.13082/) (2019)
- [BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization](https://mlsanity.com/paper/2002.03090/) (2020)
- [Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes](https://mlsanity.com/paper/2206.08852/) (2022)

## Built on by (most cited first, 3 of 3)

- [Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO](https://mlsanity.com/paper/2311.01057/) (2023)
- [Flexible and Fully Quantized Ultra-Lightweight TinyissimoYOLO for Ultra-Low-Power Edge Systems](https://mlsanity.com/paper/2307.05999/) (2023)
- [Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks](https://mlsanity.com/paper/2407.01054/) (2024)

## Abstract (arXiv)

Mixed-precision quantization, where a deep neural network's layers are quantized to different precisions, offers the opportunity to optimize the trade-offs between model size, latency, and statistical accuracy beyond what can be achieved with homogeneous-bit-width quantization. To navigate the intractable search space of mixed-precision configurations for a given network, this paper proposes a hybrid search methodology. It consists of a hardware-agnostic differentiable search algorithm followed by a hardware-aware heuristic optimization to find mixed-precision configurations latency-optimized for a specific hardware target. We evaluate our algorithm on MobileNetV1 and MobileNetV2 and deploy the resulting networks on a family of multi-core RISC-V microcontroller platforms with different hardware characteristics. We achieve up to 28.6% reduction of end-to-end latency compared to an 8-bit model at a negligible accuracy drop from a full-precision baseline on the 1000-class ImageNet dataset. We demonstrate speedups relative to an 8-bit baseline, even on systems with no hardware support for sub-byte arithmetic at negligible accuracy drop. Furthermore, we show the superiority of our approach with respect to differentiable search targeting reduced binary operation counts as a proxy for latency.