mlsanityData to 7 Oct 2026

Mitigating spectral bias for the multiscale operator learning

arXiv 2210.10890 ↗First posted 19 Oct 2022cs.LG cs.AI cs.NAcited by 43

Not on the map. Why?

Abstract (arXiv)

Neural operators have emerged as a powerful tool for learning the mapping between infinite-dimensional parameter and solution spaces of partial differential equations (PDEs). In this work, we focus on multiscale PDEs that have important applications such as reservoir modeling and turbulence prediction. We demonstrate that for such PDEs, the spectral bias towards low-frequency components presents a significant challenge for existing neural operators. To address this challenge, we propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost and encoding/decoding of multiscale solution space. We also incorporate an empirical H1 loss function to enhance the learning of high-frequency components. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 40 of its 98 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

40 of 98 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  10. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  11. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  12. 2020Rethinking Attention with Performerscited by 1,584
28 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  2. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  3. 2022Message Passing Neural PDE Solverscited by 368
  4. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  5. 2021Random Feature Attentioncited by 350
  6. 2021Choose a Transformer: Fourier or Galerkincited by 333
  7. 2020On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networkscited by 279
  8. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  9. 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
  10. 2021Factorized Fourier Neural Operatorscited by 237
  11. 2021Multiwavelet-based Operator Learning for Differential Equationscited by 229
  12. 2022U-NO: U-shaped Neural Operatorscited by 219
  13. 2019Towards Physics-informed Deep Learning for Turbulent Flow Predictioncited by 203
  14. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  15. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  16. 2021Solving and Learning Nonlinear PDEs with Gaussian Processescited by 128
  17. 2020Transformers for Modeling Physical Systemscited by 105
  18. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  19. 2022Clifford Neural Layers for PDE Modelingcited by 91
  20. 2020The Random Feature Model for Input-Output Maps between Banach Spacescited by 90
  21. 2022Learning Operators with Coupled Attentioncited by 88
  22. 2022NOMAD: Nonlinear Manifold Decoders for Operator Learningcited by 81
  23. 2022Blending Neural Operators and Relaxation Methods in PDE Numerical Solverscited by 53
  24. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  25. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  26. 2013Generalized Multiscale Finite Element Methods (GMsFEM)cited by 21
  27. 2015Multigrid with rough coefficients and Multiresolution operator decomposition from Hierarchical Information Gamescited by 15
  28. 2021Subspace Decomposition based DNN algorithm for elliptic type multi-scale PDEscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 16; 2026: 14.

43 papers.

Sort:

By area:

  1. 2023Scalable Transformer for PDE Surrogate Modelingcited by 123
  2. 2024Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularitycited by 33
  3. 2025Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systemscited by 29
  4. 2024Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspectivecited by 24
  5. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19
  6. 2024Tuning Frequency Bias of State Space Modelscited by 16
  7. 2025Recurrent Neural Operators: Stable Long-Term PDE Predictioncited by 15
  8. 2024Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flowscited by 10
  9. 2025Binned Spectral Power Loss for Improved Prediction of Chaotic Systemscited by 10
  10. 2024CaFA: Global Weather Forecasting with Factorized Attention on Spherecited by 6
  11. 2025Spectral Embedding via Chebyshev Bases for Robust DeepONet Approximationcited by 5
  12. 2024Dilated convolution neural operator for multiscale partial differential equationscited by 3
31 more
  1. 2025LaDEEP: A Deep Learning-based Surrogate Model for Large Deformation of Elastic-Plastic Solidscited by 3
  2. 2025Self-composing neural operators for high-frequency and multiscale PDE surrogatescited by 3
  3. 2024Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equationscited by 2
  4. 2024A MgNO Method for Multiphase Flow in Porous Mediacited by 2
  5. 2024M2M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations needcited by 2
  6. 2025Learn Singularly Perturbed Solutions via Homotopy Dynamicscited by 2
  7. 2025Integrating Locality-Aware Attention with Transformers for General Geometry PDEscited by 2
  8. 2025Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systemscited by 2
  9. 2026Fluids You Can Trust: Property-Preserving Operator Learning for Incompressible Flowscited by 2
  10. 2024Prediction of Multiscale Features Using Deep Learning-based Preconditioner-Solver Architecture for Darcy Equation in High-Contrast Mediacited by 1
  11. 2025SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Samplingcited by 1
  12. 2025On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fieldscited by 1
  13. 2025Breaking Scale Anchoring: Frequency Representation Learning for Accurate High-Resolution Inference from Low-Resolution Trainingcited by 1
  14. 2026Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systemscited by 1
  15. 2026Iterative Refinement Neural Operators are Learned Fixed-Point Solvers: A Principled Approach to Spectral Bias Mitigationcited by 1
  16. 2024Adaptive Error-Bounded Hierarchical Matrices for Efficient Neural Network Compressioncited by 0
  17. 2025A Greedy PDE Router for Blending Neural Operators and Classical Methodscited by 0
  18. 2025Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operatorcited by 0
  19. 2025Dual-Domain Deep Learning Method to Accelerate Local Basis Functions Computation for Reservoir Simulation in High-Contrast Porous Mediacited by 0
  20. 2025PGOT: A Physics-Geometry Operator Transformer for Complex PDEscited by 0
  21. 2026Adaptive recurrent flow map operator learning for reaction diffusion dynamicscited by 0
  22. 2026Neural Adjoint Method for Meta-optics: Accelerating Volumetric Inverse Design via Fourier Neural Operatorscited by 0
  23. 2026CATO: Charted Attention for Neural PDE Operatorscited by 0
  24. 2026Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximationscited by 0
  25. 2026McMg: A Learned Phase-Space Multi-channel Multigrid Preconditioner for Helmholtz Equationcited by 0
  26. 2026The Frame Kernel Method for Multiscale Operator Learningcited by 0
  27. 2026Spectral-Embedded Operator Learning for Three-Phase Interfacial Flow: A Ternary Cahn-Hilliard-Navier-Stokes Benchmarkcited by 0
  28. 2026Selective boundary condition reduction via learned error gatingcited by 0
  29. 2026HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometriescited by 0
  30. 2026Periodic Neural Mapping for Unsteady Rotor-Blade Pressure and Aeroelastic Load Predictioncited by 0
  31. 2026Cluster Attention Neural Operators for Solving Parametric Partial Differential Equationscited by 0