# Depth-Width Trade-offs for Neural Networks via Topological Entropy

arXiv: https://arxiv.org/abs/2010.07587 · HTML: https://mlsanity.com/paper/2010.07587/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [On the statistical complexity of quantum circuits](https://mlsanity.com/paper/2101.06154/) (2021)
- [Depth separation beyond radial functions](https://mlsanity.com/paper/2102.01621/) (2021)
- [Effects of quantum resources on the statistical complexity of quantum circuits](https://mlsanity.com/paper/2102.03282/) (2021)
- [Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis](https://mlsanity.com/paper/2205.05662/) (2022)
- [Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem](https://mlsanity.com/paper/2110.10295/) (2021)
- [Data Topology-Dependent Upper Bounds of Neural Network Widths](https://mlsanity.com/paper/2305.16375/) (2023)

## Abstract (arXiv)

One of the central problems in the study of deep learning theory is to understand how the structure properties, such as depth, width and the number of nodes, affect the expressivity of deep neural networks. In this work, we show a new connection between the expressivity of deep neural networks and topological entropy from dynamical system, which can be used to characterize depth-width trade-offs of neural networks. We provide an upper bound on the topological entropy of neural networks with continuous semi-algebraic units by the structure parameters. Specifically, the topological entropy of ReLU network with l layers and m nodes per layer is upper bounded by O(llog m). Besides, if the neural network is a good approximation of some function f, then the size of the neural network has an exponential lower bound with respect to the topological entropy of f. Moreover, we discuss the relationship between topological entropy, the number of oscillations, periods and Lipschitz constant.