mlsanityData to 7 Oct 2026

Choose a Transformer: Fourier or Galerkin

arXiv 2105.14995 ↗First posted 31 May 2021cs.LG cs.NA math.NAcited by 333

Not on the map. Why?

Abstract (arXiv)

In this paper, we apply the self-attention from the state-of-the-art Transformer in Attention Is All You Need for the first time to a data-driven operator learning problem related to partial differential equations. An effort is put together to explain the heuristics of, and to improve the efficacy of the attention mechanism. By employing the operator approximation theory in Hilbert spaces, it is demonstrated for the first time that the softmax normalization in the scaled dot-product attention is sufficient but not necessary. Without softmax, the approximation capacity of a linearized Transformer variant can be proved to be comparable to a Petrov-Galerkin projection layer-wise, and the estimate is independent with respect to the sequence length. A new layer normalization scheme mimicking the Petrov-Galerkin projection is proposed to allow a scaling to propagate through attention layers, which helps the model achieve remarkable accuracy in operator learning tasks with unnormalized data. Finally, we present three operator learning experiments, including the viscid Burgers' equation, an interface Darcy flow, and an inverse interface coefficient identification problem. The newly proposed simple attention-based operator learner, Galerkin Transformer, shows significant improvements in both training cost and evaluation accuracy over its softmax-normalized counterparts.

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). Builds on 51 of its 113 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

51 of 113 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  3. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  4. 2020On Layer Normalization in the Transformer Architecturecited by 906
  5. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  6. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  7. 2018Universal Transformerscited by 613
  8. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  9. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
  10. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  11. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  12. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  13. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  14. 2021Random Feature Attentioncited by 350
  15. 2020Multipole Graph Neural Operator for Parametric Partial Differential Equationscited by 343
  16. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  17. 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
  18. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  19. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  20. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  21. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  22. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  23. 2017Solving ill-posed inverse problems using iterative deep neural networkscited by 145
  24. 2019Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networkscited by 144
  25. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  26. 2020Understanding and mitigating gradient pathologies in physics-informed neural networkscited by 134
  27. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  28. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  29. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
  30. 2019Graph Element Networks: adaptive, structured computation and memorycited by 41
  31. 2017Adaptive Sampled Softmax with Kernel Based Samplingcited by 38
  32. 2019Equivariant Transformer Networkscited by 35
  33. 2019Sampled Softmax with Random Fourier Featurescited by 33
  34. 2019Data driven approximation of parametrized PDEs by Reduced Basis and Neural Networkscited by 21
  35. 2021Rethinking Neural Operations for Diverse Taskscited by 18
  36. 2020Implicit Kernel Attentioncited by 13
  37. 2018A probabilistic framework for multi-view feature learning with many-to-many associations via neural networkscited by 5
  38. 2016A Cheap Linear Attention Mechanism with Fast Lookups and Fixed-Size Representationscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 51; 2024: 69; 2025: 88; 2026: 103.

333 papers.

Sort:

By area:

  1. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  2. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  3. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  4. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  5. 2023Neural Operators for Accelerating Scientific Simulations and Designcited by 218
  6. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  7. 2024Poseidon: Efficient Foundation Models for PDEscited by 186
  8. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  9. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  10. 2022Clifford Neural Layers for PDE Modelingcited by 91
  11. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  12. 2022Learning Operators with Coupled Attentioncited by 88
38 more
  1. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 88
  2. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  3. 2023Solving High-Dimensional PDEs with Latent Spectral Modelscited by 84
  4. 2023A Mathematical Guide to Operator Learningcited by 74
  5. 2023Improved Operator Learning by Orthogonal Attentioncited by 58
  6. 2023Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learningcited by 57
  7. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 53
  8. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 49
  9. 2023Super-Resolution Neural Operatorcited by 48
  10. 2024CViT: Continuous Vision Transformer for Operator Learningcited by 47
  11. 2022Mitigating spectral bias for the multiscale operator learningcited by 43
  12. 2023MgNO: Efficient Parameterization of Linear Operators via Multigridcited by 40
  13. 2023Continuous-Time Functional Diffusion Processescited by 39
  14. 2024Continuum Attention for Neural Operatorscited by 39
  15. 2022How to Understand Masked Autoencoderscited by 37
  16. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  17. 2021Meta-Auto-Decoder for Solving Parametric Partial Differential Equationscited by 36
  18. 2023Domain Agnostic Fourier Neural Operatorscited by 36
  19. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  20. 2023Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equationcited by 32
  21. 2022DeepONet-Grid-UQ: A Trustworthy Deep Operator Framework for Predicting the Power Grid's Post-Fault Trajectoriescited by 30
  22. 2024On the Benefits of Memory for Modeling Time-Dependent PDEscited by 30
  23. 2023ViTO: Vision Transformer-Operatorcited by 29
  24. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
  25. 2025Generative Latent Neural PDE Solver using Flow Matchingcited by 28
  26. 2024UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptationcited by 27
  27. 2024State-space models are accurate and efficient neural operators for dynamical systemscited by 27
  28. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  29. 2025Walrus: A Cross-Domain Foundation Model for Continuum Dynamicscited by 26
  30. 2022Partial Differential Equations Meet Deep Neural Networks: A Surveycited by 25
  31. 2024Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operatorscited by 25
  32. 2022Variable-Input Deep Operator Networkscited by 24
  33. 2023Deep Operator Network Approximation Rates for Lipschitz Operatorscited by 24
  34. 2024HAMLET: Graph Transformer Neural Operator for Partial Differential Equationscited by 24
  35. 2024Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspectivecited by 24
  36. 2022Neural Integral Equationscited by 23
  37. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  38. 2023A Neural PDE Solver with Temporal Stencil Modelingcited by 22

50 most cited of 333 All 333 (JSON)