mlsanityData to 7 Oct 2026

Universal Approximation of Operators with Transformers and Neural Integral Operators

arXiv 2409.00841 ↗First posted 1 Sep 2024cs.LG cs.NA math.NAcited by 2

Not on the map. Why?

Abstract (arXiv)

We study the universal approximation properties of transformers and neural integral operators for operators in Banach spaces. In particular, we show that the transformer architecture is a universal approximator of integral operators between H"older spaces. Moreover, we show that a generalized version of neural integral operators, based on the Gavurin integral, are universal approximators of arbitrary operators between Banach spaces. Lastly, we show that a modified version of transformer, which uses Leray-Schauder mappings, is a universal approximator of operators between arbitrary Banach spaces.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

17 of 45 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  3. 2017Deep Setscited by 1,394
  4. 2021Physics-Informed Neural Operator for Learning Partial Differential Equationscited by 516
  5. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  6. 2021Choose a Transformer: Fourier or Galerkincited by 333
  7. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  8. 2020Transformers for Modeling Physical Systemscited by 105
  9. 2021Solving PDE-constrained Control Problems Using Operator Learningcited by 41
  10. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  11. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  12. 2024Universal Neural Functionalscited by 25
5 more
  1. 2022Neural Integral Equationscited by 23
  2. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  3. 2022Neural Integro-Differential Equationscited by 17
  4. 2022Attention Enables Zero Approximation Errorcited by 4
  5. 2024Projection Methods for Operator Learning and Universal Approximationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 0; 2026: 1.

2 papers.

  1. 2024Leray-Schauder Mappings for Operator Learningcited by 1
  2. 2026Neural Scaling Laws of Transformer Operator Networkcited by 0