# Mitigating spectral bias for the multiscale operator learning

arXiv: https://arxiv.org/abs/2210.10890 · HTML: https://mlsanity.com/paper/2210.10890/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 98 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks](https://mlsanity.com/paper/2012.10047/) (2020)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [Towards Physics-informed Deep Learning for Turbulent Flow Prediction](https://mlsanity.com/paper/1911.08655/) (2019)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies](https://mlsanity.com/paper/1906.00425/) (2019)
- [Solving and Learning Nonlinear PDEs with Gaussian Processes](https://mlsanity.com/paper/2103.12959/) (2021)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding](https://mlsanity.com/paper/2105.12723/) (2021)

## Built on by (most cited first, 30 of 44)

- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systems](https://mlsanity.com/paper/2503.13695/) (2025)
- [Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspective](https://mlsanity.com/paper/2404.07200/) (2024)
- [Deep Equilibrium Based Neural Operators for Steady-State PDEs](https://mlsanity.com/paper/2312.00234/) (2023)
- [Tuning Frequency Bias of State Space Models](https://mlsanity.com/paper/2410.02035/) (2024)
- [Recurrent Neural Operators: Stable Long-Term PDE Prediction](https://mlsanity.com/paper/2505.20721/) (2025)
- [Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows](https://mlsanity.com/paper/2405.17211/) (2024)
- [Binned Spectral Power Loss for Improved Prediction of Chaotic Systems](https://mlsanity.com/paper/2502.00472/) (2025)
- [CaFA: Global Weather Forecasting with Factorized Attention on Sphere](https://mlsanity.com/paper/2405.07395/) (2024)
- [Spectral Embedding via Chebyshev Bases for Robust DeepONet Approximation](https://mlsanity.com/paper/2512.09165/) (2025)
- [Dilated convolution neural operator for multiscale partial differential equations](https://mlsanity.com/paper/2408.00775/) (2024)
- [LaDEEP: A Deep Learning-based Surrogate Model for Large Deformation of Elastic-Plastic Solids](https://mlsanity.com/paper/2506.06001/) (2025)
- [Self-composing neural operators for high-frequency and multiscale PDE surrogates](https://mlsanity.com/paper/2508.20650/) (2025)
- [Fluids You Can Trust: Property-Preserving Operator Learning for Incompressible Flows](https://mlsanity.com/paper/2602.15472/) (2026)
- [Newton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations](https://mlsanity.com/paper/2405.14096/) (2024)
- [A MgNO Method for Multiphase Flow in Porous Media](https://mlsanity.com/paper/2407.02505/) (2024)
- [M2M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need](https://mlsanity.com/paper/2410.11617/) (2024)
- [Learn Singularly Perturbed Solutions via Homotopy Dynamics](https://mlsanity.com/paper/2502.00488/) (2025)
- [Integrating Locality-Aware Attention with Transformers for General Geometry PDEs](https://mlsanity.com/paper/2504.13480/) (2025)
- [Single vs. Multiple Branches in DeepONet and S-DeepONet: Network Architecture Follows Coupling in Multiphysics Systems](https://mlsanity.com/paper/2507.03660/) (2025)
- [Iterative Refinement Neural Operators are Learned Fixed-Point Solvers: A Principled Approach to Spectral Bias Mitigation](https://mlsanity.com/paper/2605.24041/) (2026)
- [Prediction of Multiscale Features Using Deep Learning-based Preconditioner-Solver Architecture for Darcy Equation in High-Contrast Media](https://mlsanity.com/paper/2411.02431/) (2024)
- [SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Sampling](https://mlsanity.com/paper/2505.04738/) (2025)
- [Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator](https://mlsanity.com/paper/2510.16816/) (2025)
- [On the accuracy of implicit neural representations for cardiovascular anatomies and hemodynamic fields](https://mlsanity.com/paper/2510.20970/) (2025)
- [Breaking Scale Anchoring: Frequency Representation Learning for Accurate High-Resolution Inference from Low-Resolution Training](https://mlsanity.com/paper/2512.05132/) (2025)
- [Multi-Scale Wavelet Transformers for Operator Learning of Dynamical Systems](https://mlsanity.com/paper/2602.01486/) (2026)
- [Adaptive Error-Bounded Hierarchical Matrices for Efficient Neural Network Compression](https://mlsanity.com/paper/2409.07028/) (2024)
- [A Greedy PDE Router for Blending Neural Operators and Classical Methods](https://mlsanity.com/paper/2509.24814/) (2025)

## Abstract (arXiv)

Neural operators have emerged as a powerful tool for learning the mapping between infinite-dimensional parameter and solution spaces of partial differential equations (PDEs). In this work, we focus on multiscale PDEs that have important applications such as reservoir modeling and turbulence prediction. We demonstrate that for such PDEs, the spectral bias towards low-frequency components presents a significant challenge for existing neural operators. To address this challenge, we propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost and encoding/decoding of multiscale solution space. We also incorporate an empirical H1 loss function to enhance the learning of high-frequency components. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.