# Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis

arXiv: https://arxiv.org/abs/2210.13300 · HTML: https://mlsanity.com/paper/2210.13300/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 148 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Universal Approximation with Deep Narrow Networks](https://mlsanity.com/paper/1905.08539/) (2019)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Wavelet neural operator: a neural operator for parametric partial differential equations](https://mlsanity.com/paper/2205.02191/) (2022)
- [Reservoir Computing Universality With Stochastic Inputs](https://mlsanity.com/paper/1807.02621/) (2018)
- [Solving the Kolmogorov PDE by means of deep learning](https://mlsanity.com/paper/1806.00421/) (2018)
- [Universal Approximation Theorems for Differentiable Geometric Deep Learning](https://mlsanity.com/paper/2101.05390/) (2021)
- [Neural Jump Ordinary Differential Equations: Consistent Continuous-Time Prediction and Filtering](https://mlsanity.com/paper/2006.04727/) (2020)
- [Fading memory echo state networks are universal](https://mlsanity.com/paper/2010.12047/) (2020)
- [Learning The Invisible: A Hybrid Deep Learning-Shearlet Framework for Limited Angle Computed Tomography](https://mlsanity.com/paper/1811.04602/) (2018)
- [Two-layer neural networks with values in a Banach space](https://mlsanity.com/paper/2105.02095/) (2021)
- [Learning the optimal Tikhonov regularizer for inverse problems](https://mlsanity.com/paper/2106.06513/) (2021)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Deep learning architectures for nonlinear operator functions and nonlinear inverse problems](https://mlsanity.com/paper/1912.11090/) (2019)
- [Deep neural networks for inverse problems with pseudodifferential operators: an application to limited-angle tomography](https://mlsanity.com/paper/2006.01620/) (2020)
- [Small Transformers Compute Universal Metric Embeddings](https://mlsanity.com/paper/2209.06788/) (2022)
- [Deep learning based numerical approximation algorithms for stochastic partial differential equations](https://mlsanity.com/paper/2012.01194/) (2020)
- [Chaotic Hedging with Iterated Integrals and Neural Networks](https://mlsanity.com/paper/2209.10166/) (2022)

## Built on by (most cited first, 22 of 22)

- [Operator Learning: Algorithms and Analysis](https://mlsanity.com/paper/2402.15715/) (2024)
- [Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equation](https://mlsanity.com/paper/2301.11509/) (2023)
- [Data Complexity Estimates for Operator Learning](https://mlsanity.com/paper/2405.15992/) (2024)
- [Mixtures of Neural Operators Reduce Active Complexity in Operator Learning](https://mlsanity.com/paper/2404.09101/) (2024)
- [A Brief Survey on the Approximation Theory for Sequence Modelling](https://mlsanity.com/paper/2302.13752/) (2023)
- [Operator Learning of Lipschitz Operators: An Information-Theoretic Perspective](https://mlsanity.com/paper/2406.18794/) (2024)
- [Reservoir kernels and Volterra series](https://mlsanity.com/paper/2212.14641/) (2022)
- [Infinite-dimensional reservoir computing](https://mlsanity.com/paper/2304.00490/) (2023)
- [Generative Neural Operators of Log-Complexity Can Simultaneously Solve Infinitely Many Convex Programs](https://mlsanity.com/paper/2508.14995/) (2025)
- [Memory of recurrent networks: Do we compute it right?](https://mlsanity.com/paper/2305.01457/) (2023)
- [Neural Snowflakes: Universal Latent Graph Inference via Trainable Latent Geometries](https://mlsanity.com/paper/2310.15003/) (2023)
- [Neural Operators Can Play Dynamic Stackelberg Games](https://mlsanity.com/paper/2411.09644/) (2024)
- [New universal operator approximation theorem for encoder-decoder architectures](https://mlsanity.com/paper/2503.24092/) (2025)
- [One model to solve them all: 2BSDE families via neural operators](https://mlsanity.com/paper/2511.01125/) (2025)
- [Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approach](https://mlsanity.com/paper/2405.20094/) (2024)
- [Simultaneously Solving Infinitely Many LQ Mean Field Games In Hilbert Spaces: The Power of Neural Operators](https://mlsanity.com/paper/2510.20017/) (2025)
- [Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts](https://mlsanity.com/paper/2402.03460/) (2024)
- [Polynomial Scaling is Possible For Neural Operator Approximations of Structured Families of BSDEs](https://mlsanity.com/paper/2410.14788/) (2024)
- [Tighter Learning Guarantees on Digital Computers via Concentration of Measure on Finite Spaces](https://mlsanity.com/paper/2402.05576/) (2024)
- [Neural Operators Can Discover Functional Clusters](https://mlsanity.com/paper/2602.23528/) (2026)
- [NeuralChaos: Optimal Adapted Approximation of Square Integrable Predictable Processes](https://mlsanity.com/paper/2607.14361/) (2026)
- [Neural operators approximate strongly continuous convex monotone semigroups](https://mlsanity.com/paper/2609.02727/) (2026)

## Abstract (arXiv)

Several non-linear operators in stochastic analysis, such as solution maps to stochastic differential equations, depend on a temporal structure which is not leveraged by contemporary neural operators designed to approximate general maps between Banach space. This paper therefore proposes an operator learning solution to this open problem by introducing a deep learning model-design framework that takes suitable infinite-dimensional linear metric spaces, e.g. Banach spaces, as inputs and returns a universal textitsequential deep learning model adapted to these linear geometries specialized for the approximation of operators encoding a temporal structure. We call these models textitCausal Neural Operators. Our main result states that the models produced by our framework can uniformly approximate on compact sets and across arbitrarily finite-time horizons H"older or smooth trace class operators, which causally map sequences between given linear metric spaces. Our analysis uncovers new quantitative relationships on the latent state-space dimension of Causal Neural Operators, which even have new implications for (classical) finite-dimensional Recurrent Neural Networks. In addition, our guarantees for recurrent neural networks are tighter than the available results inherited from feedforward neural networks when approximating dynamical systems between finite-dimensional spaces.