# Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks

arXiv: https://arxiv.org/abs/2208.07265 · HTML: https://mlsanity.com/paper/2208.07265/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 15 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Improving Neural Network Quantization without Retraining using Outlier Channel Splitting](https://mlsanity.com/paper/1901.09504/) (2019)
- [Learnable Companding Quantization for Accurate Low-bit Neural Networks](https://mlsanity.com/paper/2103.07156/) (2021)
- [ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining](https://mlsanity.com/paper/1907.07229/) (2019)
- [TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU](https://mlsanity.com/paper/2002.09481/) (2020)
- [On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations](https://mlsanity.com/paper/1703.08245/) (2017)
- [AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference](https://mlsanity.com/paper/1805.08309/) (2018)
- [ReD-CaNe: A Systematic Methodology for Resilience Analysis and Design of Capsule Networks under Approximations](https://mlsanity.com/paper/1912.00700/) (2019)
- [Efficient Bitwidth Search for Practical Mixed Precision Neural Network](https://mlsanity.com/paper/2003.07577/) (2020)
- [Deep Learning Training with Simulated Approximate Multipliers](https://mlsanity.com/paper/2001.00060/) (2019)
- [dCSR: A Memory-Efficient Sparse Matrix Representation for Parallel Neural Network Inference](https://mlsanity.com/paper/2111.12345/) (2021)
- [Evolutionary Neural Architecture Search Supporting Approximate Multipliers](https://mlsanity.com/paper/2101.11883/) (2021)

## Built on by (most cited first, 1 of 1)

- [QoS-Nets: Adaptive Approximate Neural Network Inference](https://mlsanity.com/paper/2410.07762/) (2024)

## Abstract (arXiv)

This work explores the search for heterogeneous approximate multiplier configurations for neural networks that produce high accuracy and low energy consumption. We discuss the validity of additive Gaussian noise added to accurate neural network computations as a surrogate model for behavioral simulation of approximate multipliers. The continuous and differentiable properties of the solution space spanned by the additive Gaussian noise model are used as a heuristic that generates meaningful estimates of layer robustness without the need for combinatorial optimization techniques. Instead, the amount of noise injected into the accurate computations is learned during network training using backpropagation. A probabilistic model of the multiplier error is presented to bridge the gap between the domains; the model estimates the standard deviation of the approximate multiplier error, connecting solutions in the additive Gaussian noise space to actual hardware instances. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for multiplications by 70% to 79% for different ResNet variants on the CIFAR-10 dataset with a Top-1 accuracy loss below one percentage point. For the more complex Tiny ImageNet task, our VGG16 model achieves a 53 % reduction in energy consumption with a drop in Top-5 accuracy of 0.5 percentage points. We further demonstrate that our error model can predict the parameters of an approximate multiplier in the context of the commonly used additive Gaussian noise (AGN) model with high accuracy. Our software implementation is available under https://github.com/etrommer/agn-approx.