# Solving High-Dimensional PDEs with Latent Spectral Models

arXiv: https://arxiv.org/abs/2301.12664 · HTML: https://mlsanity.com/paper/2301.12664/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Factorized Fourier Neural Operators](https://mlsanity.com/paper/2111.13802/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [Towards Physics-informed Deep Learning for Turbulent Flow Prediction](https://mlsanity.com/paper/1911.08655/) (2019)
- [Understanding and mitigating gradient pathologies in physics-informed neural networks](https://mlsanity.com/paper/2001.04536/) (2020)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Koopman neural operator as a mesh-free solver of non-linear partial differential equations](https://mlsanity.com/paper/2301.10022/) (2023)
- [KoopmanLab: machine learning for solving complex physics equations](https://mlsanity.com/paper/2301.01104/) (2023)

## Built on by (most cited first, 30 of 89)

- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Improved Operator Learning by Orthogonal Attention](https://mlsanity.com/paper/2310.12487/) (2023)
- [MgNO: Efficient Parameterization of Linear Operators via Multigrid](https://mlsanity.com/paper/2310.19809/) (2023)
- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws](https://mlsanity.com/paper/2312.11176/) (2023)
- [Beyond Regular Grids: Fourier-Based Neural Operators on Arbitrary Domains](https://mlsanity.com/paper/2305.19663/) (2023)
- [Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries](https://mlsanity.com/paper/2602.04940/) (2026)
- [PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems](https://mlsanity.com/paper/2410.01337/) (2024)
- [Harnessing Scale and Physics: A Multi-Graph Neural Operator Framework for PDEs on Arbitrary Geometries](https://mlsanity.com/paper/2411.15178/) (2024)
- [Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2510.25803/) (2025)
- [A hybrid iterative method based on MIONet for PDEs: Theory and numerical examples](https://mlsanity.com/paper/2402.07156/) (2024)
- [Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention](https://mlsanity.com/paper/2511.06294/) (2025)
- [Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows](https://mlsanity.com/paper/2405.17211/) (2024)
- [Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space](https://mlsanity.com/paper/2410.11382/) (2024)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training](https://mlsanity.com/paper/2602.20399/) (2026)
- [Turb-L1: Achieving Long-term Turbulence Tracing By Tackling Spectral Bias](https://mlsanity.com/paper/2505.19038/) (2025)
- [HelmFluid: Learning Helmholtz Dynamics for Interpretable Fluid Prediction](https://mlsanity.com/paper/2310.10565/) (2023)
- [DynST: Dynamic Sparse Training for Resource-Constrained Spatio-Temporal Forecasting](https://mlsanity.com/paper/2403.02914/) (2024)
- [M2NO: An Efficient Multi-Resolution Operator Framework for Dynamic Multi-Scale PDE Solvers](https://mlsanity.com/paper/2406.04822/) (2024)
- [SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs](https://mlsanity.com/paper/2511.18777/) (2025)
- [Spatio-Temporal Fluid Dynamics Modeling via Physical-Awareness and Parameter Diffusion Guidance](https://mlsanity.com/paper/2403.13850/) (2024)
- [PACE: Pacing Operator Learning to Accurate Optical Field Simulation for Complicated Photonic Devices](https://mlsanity.com/paper/2411.03527/) (2024)
- [FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulation](https://mlsanity.com/paper/2505.20349/) (2025)
- [Differential-Integral Neural Operator for Long-Term Turbulence Forecasting](https://mlsanity.com/paper/2509.21196/) (2025)
- [Simple yet Effective: Low-Rank Spatial Attention for Neural Operators](https://mlsanity.com/paper/2604.03582/) (2026)
- [DeepLag: Discovering Deep Lagrangian Dynamics for Intuitive Fluid Prediction](https://mlsanity.com/paper/2402.02425/) (2024)
- [Learning solution operators of PDEs defined on varying domains via MIONet](https://mlsanity.com/paper/2402.15097/) (2024)
- [CoNO: Complex Neural Operator for Continous Dynamical Physical Systems](https://mlsanity.com/paper/2406.02597/) (2024)

## Abstract (arXiv)

Deep models have achieved impressive progress in solving partial differential equations (PDEs). A burgeoning paradigm is learning neural operators to approximate the input-output mappings of PDEs. While previous deep models have explored the multiscale architectures and various operator designs, they are limited to learning the operators as a whole in the coordinate space. In real physical science problems, PDEs are complex coupled equations with numerical solvers relying on discretization into high-dimensional coordinate space, which cannot be precisely approximated by a single operator nor efficiently learned due to the curse of dimensionality. We present Latent Spectral Models (LSM) toward an efficient and precise solver for high-dimensional PDEs. Going beyond the coordinate space, LSM enables an attention-based hierarchical projection network to reduce the high-dimensional data into a compact latent space in linear time. Inspired by classical spectral methods in numerical analysis, we design a neural spectral block to solve PDEs in the latent space that approximates complex input-output mappings via learning multiple basis operators, enjoying nice theoretical guarantees for convergence and approximation. Experimentally, LSM achieves consistent state-of-the-art and yields a relative gain of 11.5% averaged on seven benchmarks covering both solid and fluid physics. Code is available at https://github.com/thuml/Latent-Spectral-Models.