# Learning Neural PDE Solvers with Parameter-Guided Channel Attention

arXiv: https://arxiv.org/abs/2304.14118 · HTML: https://mlsanity.com/paper/2304.14118/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1801.06637/) (2018)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Towards Physics-informed Deep Learning for Turbulent Flow Prediction](https://mlsanity.com/paper/1911.08655/) (2019)
- [Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks](https://mlsanity.com/paper/1905.13166/) (2019)
- [Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction](https://mlsanity.com/paper/2007.04439/) (2020)
- [Deep Fluids: A Generative Network for Parameterized Fluid Simulations](https://mlsanity.com/paper/1806.02071/) (2018)
- [Learned Coarse Models for Efficient Turbulence Simulation](https://mlsanity.com/paper/2112.15275/) (2021)
- [Learning Neural PDE Solvers with Convergence Guarantees](https://mlsanity.com/paper/1906.01200/) (2019)
- [Physical Design using Differentiable Learned Simulators](https://mlsanity.com/paper/2202.00728/) (2022)

## Built on by (most cited first, 30 of 38)

- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs](https://mlsanity.com/paper/2403.12553/) (2024)
- [Walrus: A Cross-Domain Foundation Model for Continuum Dynamics](https://mlsanity.com/paper/2511.15684/) (2025)
- [Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers](https://mlsanity.com/paper/2405.17527/) (2024)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [Active Learning for Neural PDE Solvers](https://mlsanity.com/paper/2408.01536/) (2024)
- [Zebra: In-Context Generative Pretraining for Solving Parametric PDEs](https://mlsanity.com/paper/2410.03437/) (2024)
- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations](https://mlsanity.com/paper/2406.03919/) (2024)
- [Neural Context Flows for Meta-Learning of Dynamical Systems](https://mlsanity.com/paper/2405.02154/) (2024)
- [GEPS: Boosting Generalization in Parametric PDE Neural Solvers through Adaptive Conditioning](https://mlsanity.com/paper/2410.23889/) (2024)
- [PICL: Physics Informed Contrastive Learning for Partial Differential Equations](https://mlsanity.com/paper/2401.16327/) (2024)
- [Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts](https://mlsanity.com/paper/2502.05335/) (2025)
- [Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models](https://mlsanity.com/paper/2505.10930/) (2025)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [Governing Equation Discovery from Data Based on Differential Invariants](https://mlsanity.com/paper/2505.18798/) (2025)
- [Accelerating Simulation of Two-Phase Flows with Neural PDE Surrogates](https://mlsanity.com/paper/2405.17260/) (2024)
- [Embed and Emulate: Contrastive representations for simulation-based inference](https://mlsanity.com/paper/2409.18402/) (2024)
- [A Deep Learning approach for parametrized and time dependent Partial Differential Equations using Dimensionality Reduction and Neural ODEs](https://mlsanity.com/paper/2502.08683/) (2025)
- [Ricci flow regularization in latent spaces for the forward learning of partial differential equations](https://mlsanity.com/paper/2401.14591/) (2024)
- [M2M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need](https://mlsanity.com/paper/2410.11617/) (2024)
- [Towards Generalizable PDE Dynamics Forecasting via Physics-Guided Invariant Learning](https://mlsanity.com/paper/2509.24332/) (2025)
- [Generalizing PDE Emulation with Equation-Aware Neural Operators](https://mlsanity.com/paper/2511.09729/) (2025)
- [SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems](https://mlsanity.com/paper/2405.19653/) (2024)
- [FA-INR: Adaptive Implicit Neural Representations for Interpretable Exploration of Simulation Ensembles](https://mlsanity.com/paper/2506.06858/) (2025)
- [Fast and Generalizable parameter-embedded Neural Operators for Lithium-Ion Battery Simulation](https://mlsanity.com/paper/2508.08087/) (2025)
- [Advanced Global Wildfire Activity Modeling with Hierarchical Graph ODE](https://mlsanity.com/paper/2601.01501/) (2026)
- [Generalized Neural Operator for Parametric and Boundary-Value Problems](https://mlsanity.com/paper/2607.21932/) (2026)
- [Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs](https://mlsanity.com/paper/2403.17607/) (2024)
- [Self-supervised Pretraining for Partial Differential Equations](https://mlsanity.com/paper/2407.06209/) (2024)

## Abstract (arXiv)

Scientific Machine Learning (SciML) is concerned with the development of learned emulators of physical systems governed by partial differential equations (PDE). In application domains such as weather forecasting, molecular dynamics, and inverse design, ML-based surrogate models are increasingly used to augment or replace inefficient and often non-differentiable numerical simulation algorithms. While a number of ML-based methods for approximating the solutions of PDEs have been proposed in recent years, they typically do not adapt to the parameters of the PDEs, making it difficult to generalize to PDE parameters not seen during training. We propose a Channel Attention mechanism guided by PDE Parameter Embeddings (CAPE) component for neural surrogate models and a simple yet effective curriculum learning strategy. The CAPE module can be combined with neural PDE solvers allowing them to adapt to unseen PDE parameters. The curriculum learning strategy provides a seamless transition between teacher-forcing and fully auto-regressive training. We compare CAPE in conjunction with the curriculum learning strategy using a popular PDE benchmark and obtain consistent and significant improvements over the baseline models. The experiments also show several advantages of CAPE, such as its increased ability to generalize to unseen PDE parameters without large increases inference time and parameter count.