# Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

arXiv: https://arxiv.org/abs/2007.06007 · HTML: https://mlsanity.com/paper/2007.06007/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2020.

## Built on by (most cited first, 18 of 18)

- [Achieve the Minimum Width of Neural Networks for Universal Approximation](https://mlsanity.com/paper/2209.11395/) (2022)
- [Interpolation, Approximation and Controllability of Deep Neural Networks](https://mlsanity.com/paper/2309.06015/) (2023)
- [Neural ODE Control for Trajectory Approximation of Continuity Equation](https://mlsanity.com/paper/2205.09241/) (2022)
- [Learning Robust State Observers using Neural ODEs (longer version)](https://mlsanity.com/paper/2212.00866/) (2022)
- [Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions](https://mlsanity.com/paper/2305.12205/) (2023)
- [Learning on Manifolds: Universal Approximations Properties using Geometric Controllability Conditions for Neural ODEs](https://mlsanity.com/paper/2305.08849/) (2023)
- [The Sherpa.ai Blind Vertical Federated Learning Paradigm to Minimize the Number of Communications](https://mlsanity.com/paper/2510.17901/) (2025)
- [Universal Approximation Property of Hamiltonian Deep Neural Networks](https://mlsanity.com/paper/2303.12147/) (2023)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Universal approximation property of ODENet and ResNet with a single activation function](https://mlsanity.com/paper/2410.16709/) (2024)
- [Exploring Neural Granger Causality with xLSTMs: Unveiling Temporal Dependencies in Complex Data](https://mlsanity.com/paper/2502.09981/) (2025)
- [Control, Optimal Transport and Neural Differential Equations in Supervised Learning](https://mlsanity.com/paper/2503.15105/) (2025)
- [Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations](https://mlsanity.com/paper/2605.28704/) (2026)
- [On the Topology of Neural Network Superlevel Sets](https://mlsanity.com/paper/2603.02973/) (2026)
- [Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models](https://mlsanity.com/paper/2606.31397/) (2026)
- [Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One](https://mlsanity.com/paper/2607.04597/) (2026)
- [Universal interpolation for deep residual self-attention networks](https://mlsanity.com/paper/2610.01981/) (2026)
- [Deep learning from a control vantage point](https://mlsanity.com/paper/2610.14385/) (2026)

## Abstract (arXiv)

In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property to be sufficient for an adequately deep neural network with n+1 neurons per layer to approximate arbitrarily well, on a compact set and with respect to the supremum norm, any continuous function from ℝn to ℝn. We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control systems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is to identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.