# Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks

arXiv: https://arxiv.org/abs/2206.07741 · HTML: https://mlsanity.com/paper/2206.07741/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks](https://mlsanity.com/paper/1911.03852/) (2019)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [SqueezeNext: Hardware-Aware Neural Network Design](https://mlsanity.com/paper/1803.10615/) (2018)
- [PROFIT: A Novel Training Method for sub-4-bit MobileNet Models](https://mlsanity.com/paper/2008.04693/) (2020)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks](https://mlsanity.com/paper/2102.10423/) (2021)
- [ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks](https://mlsanity.com/paper/1811.01704/) (2018)
- [Edge AI without Compromise: Efficient, Versatile and Accurate Neurocomputing in Resistive Random-Access Memory](https://mlsanity.com/paper/2108.07879/) (2021)

## Built on by (most cited first, 4 of 4)

- [Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models](https://mlsanity.com/paper/2303.08343/) (2023)
- [Mixed Precision Post Training Quantization of Neural Networks with Sensitivity Guided Search](https://mlsanity.com/paper/2302.01382/) (2023)
- [Augmenting Hessians with Inter-Layer Dependencies for Mixed-Precision Post-Training Quantization](https://mlsanity.com/paper/2306.04879/) (2023)
- [MiCoPro: End-to-End Mixed Precision HW/SW Co-design with HW-aware Proxy Model](https://mlsanity.com/paper/2608.06916/) (2026)

## Abstract (arXiv)

The large computing and memory cost of deep neural networks (DNNs) often precludes their use in resource-constrained devices. Quantizing the parameters and operations to lower bit-precision offers substantial memory and energy savings for neural network inference, facilitating the use of DNNs on edge computing platforms. Recent efforts at quantizing DNNs have employed a range of techniques encompassing progressive quantization, step-size adaptation, and gradient scaling. This paper proposes a new quantization approach for mixed precision convolutional neural networks (CNNs) targeting edge-computing. Our method establishes a new pareto frontier in model accuracy and memory footprint demonstrating a range of quantized models, delivering best-in-class accuracy below 4.3 MB of weights (wgts.) and activations (acts.). Our main contributions are: (i) hardware-aware heterogeneous differentiable quantization with tensor-sliced learned precision, (ii) targeted gradient modification for wgts. and acts. to mitigate quantization errors, and (iii) a multi-phase learning schedule to address instability in learning arising from updates to the learned quantizer and model parameters. We demonstrate the effectiveness of our techniques on the ImageNet dataset across a range of models including EfficientNet-Lite0 (e.g., 4.14MB of wgts. and acts. at 67.66% accuracy) and MobileNetV2 (e.g., 3.51MB wgts. and acts. at 65.39% accuracy).