# WRPN: Wide Reduced-Precision Networks

arXiv: https://arxiv.org/abs/1709.01134 · HTML: https://mlsanity.com/paper/1709.01134/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2017.

## Built on by (most cited first, 30 of 51)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution](https://mlsanity.com/paper/2106.02295/) (2021)
- [CPT: Efficient Deep Neural Network Training via Cyclic Precision](https://mlsanity.com/paper/2101.09868/) (2021)
- [Lightweight compression of neural network feature tensors for collaborative intelligence](https://mlsanity.com/paper/2105.06002/) (2021)
- [FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://mlsanity.com/paper/2407.07093/) (2024)
- [RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy](https://mlsanity.com/paper/2103.13813/) (2021)
- [Q-LIC: Quantizing Learned Image Compression with Channel Splitting](https://mlsanity.com/paper/2205.14510/) (2022)
- [Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity](https://mlsanity.com/paper/2309.06626/) (2023)
- [Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence](https://mlsanity.com/paper/2105.07102/) (2021)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation](https://mlsanity.com/paper/2102.07444/) (2021)
- [BoolNet: Minimizing The Energy Consumption of Binary Neural Networks](https://mlsanity.com/paper/2106.06991/) (2021)
- [MBQuant: A Novel Multi-Branch Topology Method for Arbitrary Bit-width Network Quantization](https://mlsanity.com/paper/2305.08117/) (2023)
- [NullaNet Tiny: Ultra-low-latency DNN Inference Through Fixed-function Combinational Logic](https://mlsanity.com/paper/2104.05421/) (2021)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [End-to-End Learned Image Compression with Quantized Weights and Activations](https://mlsanity.com/paper/2111.09348/) (2021)
- [Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensing](https://mlsanity.com/paper/2309.06047/) (2023)
- [What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias](https://mlsanity.com/paper/2410.08407/) (2024)
- [Activation Density based Mixed-Precision Quantization for Energy Efficient Neural Networks](https://mlsanity.com/paper/2101.04354/) (2021)
- [On-Device Learning with Binary Neural Networks](https://mlsanity.com/paper/2308.15308/) (2023)

## Abstract (arXiv)

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.