# On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

arXiv: https://arxiv.org/abs/2210.15279 · HTML: https://mlsanity.com/paper/2210.15279/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 27 Oct 2022.

## Builds on (most cited first, 19 of 19)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Deep Complex Networks](https://mlsanity.com/paper/1705.09792/) (2017)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [A Survey of Complex-Valued Neural Networks](https://mlsanity.com/paper/2101.12249/) (2021)
- [Universal approximations of permutation invariant/equivariant functions by deep neural networks](https://mlsanity.com/paper/1903.01939/) (2019)
- [The universal approximation theorem for complex-valued neural networks](https://mlsanity.com/paper/2012.03351/) (2020)
- [Evaluation of Complex-Valued Neural Networks on Real-Valued Classification Tasks](https://mlsanity.com/paper/1811.12351/) (2018)
- [Deep Neural Network Approximation of Invariant Functions through Dynamical Systems](https://mlsanity.com/paper/2208.08707/) (2022)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [Towards Understanding Theoretical Advantages of Complex-Reaction Networks](https://mlsanity.com/paper/2108.06711/) (2021)
- [Flexible Transmitter Network](https://mlsanity.com/paper/2004.03839/) (2020)
- [Neural Network Gaussian Processes by Increasing Depth](https://mlsanity.com/paper/2108.12862/) (2021)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)

## Abstract (arXiv)

Recent years have witnessed a hot wave of deep neural networks in various domains; however, it is not yet well understood theoretically. A theoretical characterization of deep neural networks should point out their approximation ability and complexity, i.e., showing which architecture and size are sufficient to handle the concerned tasks. This work takes one step on this direction by theoretically studying the approximation and complexity of deep neural networks to invariant functions. We first prove that the invariant functions can be universally approximated by deep neural networks. Then we show that a broad range of invariant functions can be asymptotically approximated by various types of neural network models that includes the complex-valued neural networks, convolutional neural networks, and Bayesian neural networks using a polynomial number of parameters or optimization iterations. We also provide a feasible application that connects the parameter estimation and forecasting of high-resolution signals with our theoretical conclusions. The empirical results obtained on simulation experiments demonstrate the effectiveness of our method.