mlsanityData to 7 Oct 2026

Continuum Attention for Neural Operators

arXiv 2406.06486 ↗First posted 10 Jun 2024cs.LG cs.NA math.NAcited by 39

Not on the map. Why?

Abstract (arXiv)

Transformers, and the attention mechanism in particular, have become ubiquitous in machine learning. Their success in modeling nonlocal, long-range correlations has led to their widespread adoption in natural language processing, computer vision, and time series problems. Neural operators, which map spaces of functions into spaces of functions, are necessarily both nonlinear and nonlocal if they are universal; it is thus natural to ask whether the attention mechanism can be used in the design of neural operators. Motivated by this, we study transformers in the function space setting. We formulate attention as a map between infinite dimensional function spaces and prove that the attention mechanism as implemented in practice is a Monte Carlo or finite difference approximation of this operator. The function space formulation allows for the design of transformer neural operators, a class of architectures designed to learn mappings between function spaces. In this paper, we state and prove the first universal approximation result for transformer neural operators, using only a slight modification of the architecture implemented in practice. The prohibitive cost of applying the attention operator to functions defined on multi-dimensional domains leads to the need for more efficient attention-based architectures. For this reason we also introduce a function space generalization of the patching strategy from computer vision, and introduce a class of associated neural operators. Numerical results, on an array of operator learning problems, demonstrate the promise of our approaches to function space formulations of attention and their use in neural operators.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

24 of 36 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2022Transformers in Time Series: A Surveycited by 489
  10. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  11. 2021Choose a Transformer: Fourier or Galerkincited by 333
  12. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
12 more
  1. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  2. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  3. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  4. 2023The emergence of clusters in self-attention dynamicscited by 133
  5. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
  6. 2024Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEscited by 58
  7. 2023Nonlocality and Nonlinearity Implies Universality in Operator Learningcited by 43
  8. 2020Sparse and Continuous Attention Mechanismscited by 37
  9. 2023ViTO: Vision Transformer-Operatorcited by 29
  10. 2021Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machinescited by 22
  11. 2018Inverse Problems and Data Assimilationcited by 18
  12. 2021Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformerscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 28.

39 papers.

Sort:

  1. 2025Active operator learning with predictive uncertainty quantification for partial differential equationscited by 12
  2. 2025Learning Enhanced Ensemble Filterscited by 11
  3. 2025Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learningcited by 11
  4. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  5. 2025Operator Learning at Machine Precisioncited by 6
  6. 2024Neural Operators Can Play Dynamic Stackelberg Gamescited by 5
  7. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  8. 2026Performance of Neural and Polynomial Operator Surrogatescited by 5
  9. 2025Continuum Transformers Perform In-Context Learning by Operator Gradient Descentcited by 3
  10. 2025Method of Manufactured Learning for Solver-free Training of Neural Operatorscited by 3
  11. 2025Learning High-dimensional Ionic Model Dynamics Using Fourier Neural Operatorscited by 2
  12. 2026One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operatorscited by 2
27 more
  1. 2026Function graph transformers universally approximate operators between function spacescited by 2
  2. 2026Topological Neural Operatorscited by 2
  3. 2026A Physics-Informed Fourier-Wavelet Transformer for Multiscale Computational Fluid Dynamics Surrogate Modelingcited by 2
  4. 2025Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operatorscited by 1
  5. 2026Adaptive Physics Transformer with Fused Global-Local Attention for Subsurface Energy Systemscited by 1
  6. 2026Inverse Neural Operator for ODE Parameter Optimizationcited by 1
  7. 2026Operator Learning for Smoothing and Forecastingcited by 1
  8. 2026Anchored-Branched Steady-state WInd Flow Transformer (AB-SWIFT): a metamodel for 3D atmospheric flow in urban environmentscited by 1
  9. 2026A meshfree exterior calculus for generalizable and data-efficient learning of physics from point cloudscited by 1
  10. 2026Zero-shot generalization of transformer neural operators to larger domainscited by 1
  11. 2026A Theory of Diversity for Random Matrices with Applications to In-Context Learning of Schr"odinger Equationscited by 0
  12. 2026Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamicscited by 0
  13. 2026Prior-Fitted Functional Flow: In-Context Generative Models for Pharmacokineticscited by 0
  14. 2026Learning Neural Operator Surrogates for the Black Hole Accretion Codecited by 0
  15. 2026NOFE - Neural Operator Function Embeddingcited by 0
  16. 2026Revisiting Neural Processes via Fourier Transform and Volterra Seriescited by 0
  17. 2026Geometry-Aware Anisotropic Boundary Correction for Aerodynamic Simulationcited by 0
  18. 2026Temperature Field Reconstruction of Tungsten Monoblock Divertor on EAST using Physics-aware Neural Operator Transformercited by 0
  19. 2026An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformerscited by 0
  20. 2026Deep operator learning for efficient sampling from invariant measures of stochastic differential equationscited by 0
  21. 2026Cost-Accuracy Trade-offs: Neural Operator vs Classical Numerical Solvercited by 0
  22. 2026Refinement Symmetry in Multimodal Transformerscited by 0
  23. 2026Neural Scaling Laws of Transformer Operator Networkcited by 0
  24. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0
  25. 2026CANTO: CAD-Native Transformer Operators for AI-Aided Engineeringcited by 0
  26. 2026Function-Space Transformer with Adaptive Anchorscited by 0
  27. 2026Attention Kernels for Learning Maps Between Heavy-Tailed Measurescited by 0