# Improved Operator Learning by Orthogonal Attention

arXiv: https://arxiv.org/abs/2310.12487 · HTML: https://mlsanity.com/paper/2310.12487/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 19 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [Solving High-Dimensional PDEs with Latent Spectral Models](https://mlsanity.com/paper/2301.12664/) (2023)
- [Koopman neural operator as a mesh-free solver of non-linear partial differential equations](https://mlsanity.com/paper/2301.10022/) (2023)
- [ViTO: Vision Transformer-Operator](https://mlsanity.com/paper/2303.08891/) (2023)
- [NeuralEF: Deconstructing Kernels by Deep Neural Networks](https://mlsanity.com/paper/2205.00165/) (2022)
- [Neural Eigenfunctions Are Structured Representation Learners](https://mlsanity.com/paper/2210.12637/) (2022)
- [Continuous Spatiotemporal Transformers](https://mlsanity.com/paper/2301.13338/) (2023)

## Built on by (most cited first, 30 of 61)

- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers](https://mlsanity.com/paper/2502.09692/) (2025)
- [Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems](https://mlsanity.com/paper/2502.17019/) (2025)
- [Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries](https://mlsanity.com/paper/2602.04940/) (2026)
- [Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning](https://mlsanity.com/paper/2405.09285/) (2024)
- [Harnessing Scale and Physics: A Multi-Graph Neural Operator Framework for PDEs on Arbitrary Geometries](https://mlsanity.com/paper/2411.15178/) (2024)
- [Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention](https://mlsanity.com/paper/2511.06294/) (2025)
- [Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows](https://mlsanity.com/paper/2405.17211/) (2024)
- [Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space](https://mlsanity.com/paper/2410.11382/) (2024)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [M2NO: An Efficient Multi-Resolution Operator Framework for Dynamic Multi-Scale PDE Solvers](https://mlsanity.com/paper/2406.04822/) (2024)
- [Neural Green's Functions](https://mlsanity.com/paper/2511.01924/) (2025)
- [SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs](https://mlsanity.com/paper/2511.18777/) (2025)
- [FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulation](https://mlsanity.com/paper/2505.20349/) (2025)
- [DiffusionRollout: Uncertainty-Aware Rollout Planning in Long-Horizon PDE Solving](https://mlsanity.com/paper/2602.13616/) (2026)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [DiffFluid: Plain Diffusion Models are Effective Predictors of Flow Dynamics](https://mlsanity.com/paper/2409.13665/) (2024)
- [Simple yet Effective: Low-Rank Spatial Attention for Neural Operators](https://mlsanity.com/paper/2604.03582/) (2026)
- [LaDEEP: A Deep Learning-based Surrogate Model for Large Deformation of Elastic-Plastic Solids](https://mlsanity.com/paper/2506.06001/) (2025)
- [MSPT: Efficient Large-Scale Physical Modeling via Parallelized Multi-Scale Attention](https://mlsanity.com/paper/2512.01738/) (2025)
- [Topology-Preserving Neural Operator Learning via Hodge Decomposition](https://mlsanity.com/paper/2605.13834/) (2026)
- [Integrating Locality-Aware Attention with Transformers for General Geometry PDEs](https://mlsanity.com/paper/2504.13480/) (2025)
- [GeoMaNO: Geometric Mamba Neural Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.12020/) (2025)
- [MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving](https://mlsanity.com/paper/2506.17230/) (2025)
- [GFocal: A Global-Focal Neural Operator for Solving PDEs on Arbitrary Geometries](https://mlsanity.com/paper/2508.04463/) (2025)
- [From Basis to Basis: Gaussian Particle Representation for Interpretable PDE Operators](https://mlsanity.com/paper/2602.21551/) (2026)
- [Di-BiLPS: Denoising induced Bidirectional Latent-PDE-Solver under Sparse Observations](https://mlsanity.com/paper/2605.13790/) (2026)
- [SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Sampling](https://mlsanity.com/paper/2505.04738/) (2025)
- [Graph-Based Operator Learning from Limited Data on Irregular Domains](https://mlsanity.com/paper/2505.18923/) (2025)
- [BSA: Ball Sparse Attention for Large-scale Geometries](https://mlsanity.com/paper/2506.12541/) (2025)

## Abstract (arXiv)

Neural operators, as an efficient surrogate model for learning the solutions of PDEs, have received extensive attention in the field of scientific machine learning. Among them, attention-based neural operators have become one of the mainstreams in related research. However, existing approaches overfit the limited training data due to the considerable number of parameters in the attention mechanism. To address this, we develop an orthogonal attention based on the eigendecomposition of the kernel integral operator and the neural approximation of eigenfunctions. The orthogonalization naturally poses a proper regularization effect on the resulting neural operator, which aids in resisting overfitting and boosting generalization. Experiments on six standard neural operator benchmark datasets comprising both regular and irregular geometries show that our method can outperform competing baselines with decent margins.