# Universal Approximation Theorems for Differentiable Geometric Deep Learning

arXiv: https://arxiv.org/abs/2101.05390 · HTML: https://mlsanity.com/paper/2101.05390/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 138 references that are in the map.
- First posted on arXiv 13 Jan 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Structural-RNN: Deep Learning on Spatio-Temporal Graphs](https://mlsanity.com/paper/1511.05298/) (2015)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Riemannian Adaptive Optimization Methods](https://mlsanity.com/paper/1810.00760/) (2018)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Universal approximations of invariant maps by neural networks](https://mlsanity.com/paper/1804.10306/) (2018)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Error bounds for approximations with deep ReLU neural networks in Ws,p norms](https://mlsanity.com/paper/1902.07896/) (2019)
- [Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation](https://mlsanity.com/paper/1603.03236/) (2016)
- [The phase diagram of approximation rates for deep neural networks](https://mlsanity.com/paper/1906.09477/) (2019)
- [Trivializations for Gradient-Based Optimization on Manifolds](https://mlsanity.com/paper/1909.09501/) (2019)
- [geomstats: a Python Package for Riemannian Geometry in Machine Learning](https://mlsanity.com/paper/1805.08308/) (2018)
- [Theano: A Python framework for fast computation of mathematical expressions](https://mlsanity.com/paper/1605.02688/) (2016)
- [Approximation spaces of deep neural networks](https://mlsanity.com/paper/1905.01208/) (2019)
- [Sparse Gaussian Processes with Spherical Harmonic Features](https://mlsanity.com/paper/2006.16649/) (2020)
- [Deep Learning on Lie Groups for Skeleton-based Action Recognition](https://mlsanity.com/paper/1612.05877/) (2016)
- [Non-Euclidean Universal Approximation](https://mlsanity.com/paper/2006.02341/) (2020)

## Built on by (most cited first, 30 of 41)

- [Transformers are Universal In-context Learners](https://mlsanity.com/paper/2408.01367/) (2024)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Localized Zeroth-Order Prompt Optimization](https://mlsanity.com/paper/2403.02993/) (2024)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis](https://mlsanity.com/paper/2210.13300/) (2022)
- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [A Survey on Universal Approximation Theorems](https://mlsanity.com/paper/2407.12895/) (2024)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [Energy-Guided Continuous Entropic Barycenter Estimation for General Costs](https://mlsanity.com/paper/2310.01105/) (2023)
- [Universal approximation results for neural networks with non-polynomial activation function over non-compact domains](https://mlsanity.com/paper/2410.14759/) (2024)
- [Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?](https://mlsanity.com/paper/2204.11231/) (2022)
- [Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries](https://mlsanity.com/paper/2310.15003/) (2023)
- [Total Energy Shaping with Neural Interconnection and Damping Assignment -- Passivity Based Control](https://mlsanity.com/paper/2112.12999/) (2021)
- [Generalized Permutants and Graph GENEOs](https://mlsanity.com/paper/2206.14798/) (2022)
- [Neural Operators Can Play Dynamic Stackelberg Games](https://mlsanity.com/paper/2411.09644/) (2024)
- [Universal Regular Conditional Distributions](https://mlsanity.com/paper/2105.07743/) (2021)
- [Capacity Bounds for Hyperbolic Neural Network Representations of Latent Tree Structures](https://mlsanity.com/paper/2308.09250/) (2023)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [Approximation Rates in Besov Norms and Sample-Complexity of Kolmogorov-Arnold Networks with Residual Connections](https://mlsanity.com/paper/2504.15110/) (2025)
- [Shape-Informed Clustering of Multi-Dimensional Functional Data via Deep Functional Autoencoders](https://mlsanity.com/paper/2509.22969/) (2025)
- [Universal approximation with complex-valued deep narrow neural networks](https://mlsanity.com/paper/2305.16910/) (2023)
- [Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approach](https://mlsanity.com/paper/2405.20094/) (2024)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Approximation properties of neural ODEs](https://mlsanity.com/paper/2503.15696/) (2025)
- [Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data](https://mlsanity.com/paper/2509.00924/) (2025)
- [Neural Network Approximation of Continuous Functions in High Dimensions with Applications to Inverse Problems](https://mlsanity.com/paper/2208.13305/) (2022)
- [Keep It Light! Simplifying Image Clustering Via Text-Free Adapters](https://mlsanity.com/paper/2502.04226/) (2025)
- [Understanding the role of autoencoders for stiff dynamical systems using information theory](https://mlsanity.com/paper/2503.06325/) (2025)
- [Autonomous-Flow-Based Generation](https://mlsanity.com/paper/2511.09902/) (2025)
- [Upper Approximation Bounds for Neural Oscillators](https://mlsanity.com/paper/2512.01015/) (2025)

## Abstract (arXiv)

This paper addresses the growing need to process non-Euclidean data, by introducing a geometric deep learning (GDL) framework for building universal feedforward-type models compatible with differentiable manifold geometries. We show that our GDL models can approximate any continuous target function uniformly on compact sets of a controlled maximum diameter. We obtain curvature-dependent lower-bounds on this maximum diameter and upper-bounds on the depth of our approximating GDL models. Conversely, we find that there is always a continuous function between any two non-degenerate compact manifolds that any "locally-defined" GDL model cannot uniformly approximate. Our last main result identifies data-dependent conditions guaranteeing that the GDL model implementing our approximation breaks "the curse of dimensionality." We find that any "real-world" (i.e. finite) dataset always satisfies our condition and, conversely, any dataset satisfies our requirement if the target function is smooth. As applications, we confirm the universal approximation capabilities of the following GDL models: Ganea et al. (2018)'s hyperbolic feedforward networks, the architecture implementing Krishnan et al. (2015)'s deep Kalman-Filter, and deep softmax classifiers. We build universal extensions/variants of: the SPD-matrix regressor of Meyer et al. (2011), and Fletcher (2003)'s Procrustean regressor. In the Euclidean setting, our results imply a quantitative version of Kidger and Lyons (2020)'s approximation theorem and a data-dependent version of Yarotsky and Zhevnerchuk (2019)'s uncursed approximation rates.