# The Upper Bound on Knots in Neural Networks

arXiv: https://arxiv.org/abs/1611.09448 · HTML: https://mlsanity.com/paper/1611.09448/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2016.

## Built on by (most cited first, 1 of 1)

- [Free-Knots Kolmogorov-Arnold Network: On the Analysis of Spline Knots and Advancing Stability](https://mlsanity.com/paper/2501.09283/) (2025)

## Abstract (arXiv)

Neural networks with rectified linear unit activations are essentially multivariate linear splines. As such, one of many ways to measure the "complexity" or "expressivity" of a neural network is to count the number of knots in the spline model. We study the number of knots in fully-connected feedforward neural networks with rectified linear unit activation functions. We intentionally keep the neural networks very simple, so as to make theoretical analyses more approachable. An induction on the number of layers l reveals a tight upper bound on the number of knots in ℝ → ℝp deep neural networks. With ni gg 1 neurons in layer i = 1, dots, l, the upper bound is approximately n1 dots nl. We then show that the exact upper bound is tight, and we demonstrate the upper bound with an example. The purpose of these analyses is to pave a path for understanding the behavior of general ℝq → ℝp neural networks.