# Computational Complexity Evaluation of Neural Network Applications in Signal Processing

arXiv: https://arxiv.org/abs/2206.12191 · HTML: https://mlsanity.com/paper/2206.12191/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 119 references that are in the map.
- First posted on arXiv 24 Jun 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [1D Convolutional Neural Networks and Applications: A Survey](https://mlsanity.com/paper/1905.03554/) (2019)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Any-Precision Deep Neural Networks](https://mlsanity.com/paper/1911.07346/) (2019)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions](https://mlsanity.com/paper/1806.09228/) (2018)
- [Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference](https://mlsanity.com/paper/2102.11289/) (2021)
- [Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks](https://mlsanity.com/paper/2203.05025/) (2022)
- [Hardware-oriented Approximation of Convolutional Neural Networks](https://mlsanity.com/paper/1604.03168/) (2016)
- [Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems](https://mlsanity.com/paper/2103.08212/) (2021)
- [Deep Learning with Dynamic Computation Graphs](https://mlsanity.com/paper/1702.02181/) (2017)
- [Compact and Computationally Efficient Representation of Deep Neural Networks](https://mlsanity.com/paper/1805.10692/) (2018)
- [A Review of Designs and Applications of Echo State Networks](https://mlsanity.com/paper/2012.02974/) (2020)
- [On the Statistical Challenges of Echo State Networks and Some Potential Remedies](https://mlsanity.com/paper/1802.07369/) (2018)

## Built on by (most cited first, 1 of 1)

- [Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation](https://mlsanity.com/paper/2408.02685/) (2024)

## Abstract (arXiv)

In this paper, we provide a systematic approach for assessing and comparing the computational complexity of neural network layers in digital signal processing. We provide and link four software-to-hardware complexity measures, defining how the different complexity metrics relate to the layers' hyper-parameters. This paper explains how to compute these four metrics for feed-forward and recurrent layers, and defines in which case we ought to use a particular metric depending on whether we characterize a more soft- or hardware-oriented application. One of the four metrics, called `the number of additions and bit shifts (NABS)', is newly introduced for heterogeneous quantization. NABS characterizes the impact of not only the bitwidth used in the operation but also the type of quantization used in the arithmetical operations. We intend this work to serve as a baseline for the different levels (purposes) of complexity estimation related to the neural networks' application in real-time digital signal processing, aiming at unifying the computational complexity estimation.