# PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks

arXiv: https://arxiv.org/abs/2404.00103 · HTML: https://mlsanity.com/paper/2404.00103/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Low-bit Quantization of Neural Networks for Efficient Inference](https://mlsanity.com/paper/1902.06822/) (2019)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [PROFIT: A Novel Training Method for sub-4-bit MobileNet Models](https://mlsanity.com/paper/2008.04693/) (2020)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?](https://mlsanity.com/paper/2001.05936/) (2020)
- [FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations](https://mlsanity.com/paper/2012.12206/) (2020)
- [Pareto-Optimal Quantized ResNet Is Mostly 4-bit](https://mlsanity.com/paper/2105.03536/) (2021)
- [Binarizing MobileNet via Evolution-based Searching](https://mlsanity.com/paper/2005.06305/) (2020)
- [Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks](https://mlsanity.com/paper/1612.03940/) (2016)
- [MoBiNet: A Mobile Binary Network for Image Classification](https://mlsanity.com/paper/1907.12629/) (2019)

## Built on by (most cited first, 4 of 4)

- [Robust Training of Neural Networks at Arbitrary Precision and Sparsity](https://mlsanity.com/paper/2409.09245/) (2024)
- [LogicIR: Logic Gate Networks for Image Restoration](https://mlsanity.com/paper/2606.26609/) (2026)
- [SIRA: Scaled-Integer Range Analysis for Optimizing FPGA Dataflow Neural Network Accelerators](https://mlsanity.com/paper/2508.21493/) (2025)
- [Overcoming Kernel Redundancy for Scaling Logic Gate Networks](https://mlsanity.com/paper/2610.01069/) (2026)

## Abstract (arXiv)

Low-precision quantization is recognized for its efficacy in neural network optimization. Our analysis reveals that non-quantized elementwise operations which are prevalent in layers such as parameterized activation functions, batch normalization, and quantization scaling dominate the inference cost of low-precision models. These non-quantized elementwise operations are commonly overlooked in SOTA efficiency metrics such as Arithmetic Computation Effort (ACE). In this paper, we propose ACEv2 - an extended version of ACE which offers a better alignment with the inference cost of quantized models and their energy consumption on ML hardware. Moreover, we introduce PikeLPN, a model that addresses these efficiency issues by applying quantization to both elementwise operations and multiply-accumulate operations. In particular, we present a novel quantization technique for batch normalization layers named QuantNorm which allows for quantizing the batch normalization parameters without compromising the model performance. Additionally, we propose applying Double Quantization where the quantization scaling parameters are quantized. Furthermore, we recognize and resolve the issue of distribution mismatch in Separable Convolution layers by introducing Distribution-Heterogeneous Quantization which enables quantizing them to low-precision. PikeLPN achieves Pareto-optimality in efficiency-accuracy trade-off with up to 3X efficiency improvement compared to SOTA low-precision models.