mlsanityData to 7 Oct 2026

Improved Operator Learning by Orthogonal Attention

arXiv 2310.12487 ↗First posted 19 Oct 2023cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

27 of 40 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2018Image Transformercited by 770
  11. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  12. 2022Fourier Neural Operator with Learned Deformations for PDEs on General Geometriescited by 416
15 more
  1. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  2. 2021Choose a Transformer: Fourier or Galerkincited by 333
  3. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  4. 2021U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flowcited by 292
  5. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  6. 2021Factorized Fourier Neural Operatorscited by 237
  7. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 229
  8. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  9. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  10. 2023Solving High-Dimensional PDEs with Latent Spectral Modelscited by 84
  11. 2023Koopman neural operator as a mesh-free solver of non-linear partial differential equationscited by 29
  12. 2023ViTO: Vision Transformer-Operatorcited by 29
  13. 2022NeuralEF: Deconstructing Kernels by Deep Neural Networkscited by 26
  14. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  15. 2023Continuous Spatiotemporal Transformerscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 24; 2026: 25.

58 papers.

Sort:

By area:

  1. 2024Transolver: A Fast Transformer Solver for PDEs on General Geometriescited by 299
  2. 2025AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformerscited by 49
  3. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 21
  4. 2026Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometriescited by 20
  5. 2024Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learningcited by 17
  6. 2024Harnessing Scale and Physics: A Multi-Graph Neural Operator Framework for PDEs on Arbitrary Geometriescited by 15
  7. 2025Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attentioncited by 11
  8. 2024Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flowscited by 10
  9. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  10. 2025Latent Mamba Operator for Partial Differential Equationscited by 8
  11. 2024M2NO: An Efficient Multi-Resolution Operator Framework for Dynamic Multi-Scale PDE Solverscited by 6
  12. 2025Neural Green's Functionscited by 6
38 more
  1. 2025SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEscited by 6
  2. 2024Spectral Convolutional Conditional Neural Processescited by 4
  3. 2024DiffFluid: Plain Diffusion Models are Effective Predictors of Flow Dynamicscited by 4
  4. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 4
  5. 2026DiffusionRollout: Uncertainty-Aware Rollout Planning in Long-Horizon PDE Solvingcited by 4
  6. 2026Simple yet Effective: Low-Rank Spatial Attention for Neural Operatorscited by 4
  7. 2025LaDEEP: A Deep Learning-based Surrogate Model for Large Deformation of Elastic-Plastic Solidscited by 3
  8. 2025MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attentioncited by 3
  9. 2025Integrating Locality-Aware Attention with Transformers for General Geometry PDEscited by 2
  10. 2025GeoMaNO: Geometric Mamba Neural Operator for Partial Differential Equationscited by 2
  11. 2025MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solvingcited by 2
  12. 2025GFocal: A Global-Focal Neural Operator for Solving PDEs on Arbitrary Geometriescited by 2
  13. 2026From Basis to Basis: Gaussian Particle Representation for Interpretable PDE Operatorscited by 2
  14. 2025SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Samplingcited by 1
  15. 2025Graph-Based Operator Learning from Limited Data on Irregular Domainscited by 1
  16. 2025BSA: Ball Sparse Attention for Large-scale Geometriescited by 1
  17. 2025Accelerating PDE-Constrained Optimization by the Derivative of Neural Operatorscited by 1
  18. 2025Can Data-Driven Dynamics Reveal Hidden Physics? There Is A Need for Interpretable Neural Operatorscited by 1
  19. 2026Bridging Spectral Operator Learning and U-Net Hierarchies: SpectraNet for Stable Autoregressive PDE Surrogatescited by 1
  20. 2026Di-BiLPS: Denoising induced Bidirectional Latent-PDE-Solver under Sparse Observationscited by 1
  21. 2026Topology-Preserving Neural Operator Learning via Hodge Decompositioncited by 1
  22. 2024Deep Parallel Spectral Neural Operators for Solving Partial Differential Equations with Enhanced Low-Frequency Learning Capabilitycited by 0
  23. 2025SUPRA: Subspace Parameterized Attention for Neural Operator on General Domainscited by 0
  24. 2025Reformulating Neural Operators in d+1 Dimensions for Embedding Evolutioncited by 0
  25. 2025Ultrasound Tomography of Musculoskeletal Tissues with Generative Neural Physicscited by 0
  26. 2025Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operatorcited by 0
  27. 2025PIBNet: a Physics-Inspired Boundary Network for Multiple Scattering Simulationscited by 0
  28. 2025PGOT: A Physics-Geometry Operator Transformer for Complex PDEscited by 0
  29. 2026Learning Neural Operators from Partial Observations via Latent Autoregressive Modelingcited by 0
  30. 2026Efficient Dilated Squeeze and Excitation Neural Operator for Differential Equationscited by 0
  31. 2026From Complex Dynamics to DynFormer: Rethinking Transformers for PDEscited by 0
  32. 2026Linear-Nonlinear Fusion Neural Operator for Partial Differential Equationscited by 0
  33. 2026FlowForge: A Staged Local Rollout Engine for Flow-Field Predictioncited by 0
  34. 2026CATO: Charted Attention for Neural PDE Operatorscited by 0
  35. 2026Functional Attention: From Pairwise Affinities to Functional Correspondencescited by 0
  36. 2026TF-SNO: Time-Frequency Gated Spectral Neural Operators for Learning Non-Stationary Partial Differential Equationscited by 0
  37. 2026Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEscited by 0
  38. 2026GAIA: Geometry-Adaptive Operator Learning for Forward and Inverse Problemscited by 0

50 most cited of 58 All 58 (JSON)