# Universal Approximation of Operators with Transformers and Neural Integral Operators

arXiv: https://arxiv.org/abs/2409.00841 · HTML: https://mlsanity.com/paper/2409.00841/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Solving PDE-constrained Control Problems Using Operator Learning](https://mlsanity.com/paper/2111.04941/) (2021)
- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input](https://mlsanity.com/paper/2305.18699/) (2023)
- [Universal Neural Functionals](https://mlsanity.com/paper/2402.05232/) (2024)
- [Neural Integral Equations](https://mlsanity.com/paper/2209.15190/) (2022)
- [Transformer Meets Boundary Value Inverse Problems](https://mlsanity.com/paper/2209.14977/) (2022)
- [Neural Integro-Differential Equations](https://mlsanity.com/paper/2206.14282/) (2022)
- [Attention Enables Zero Approximation Error](https://mlsanity.com/paper/2202.12166/) (2022)
- [Projection Methods for Operator Learning and Universal Approximation](https://mlsanity.com/paper/2406.12264/) (2024)

## Built on by (most cited first, 2 of 2)

- [Leray-Schauder Mappings for Operator Learning](https://mlsanity.com/paper/2410.01746/) (2024)
- [Neural Scaling Laws of Transformer Operator Network](https://mlsanity.com/paper/2609.33533/) (2026)

## Abstract (arXiv)

We study the universal approximation properties of transformers and neural integral operators for operators in Banach spaces. In particular, we show that the transformer architecture is a universal approximator of integral operators between H"older spaces. Moreover, we show that a generalized version of neural integral operators, based on the Gavurin integral, are universal approximators of arbitrary operators between Banach spaces. Lastly, we show that a modified version of transformer, which uses Leray-Schauder mappings, is a universal approximator of operators between arbitrary Banach spaces.