# Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers

arXiv: https://arxiv.org/abs/2405.17527 · HTML: https://mlsanity.com/paper/2405.17527/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Towards Multi-spatiotemporal-scale Generalized PDE Modeling](https://mlsanity.com/paper/2209.15616/) (2022)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [In-Context Operator Learning with Data Prompts for Differential Equation Problems](https://mlsanity.com/paper/2304.07993/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Learning Neural PDE Solvers with Parameter-Guided Channel Attention](https://mlsanity.com/paper/2304.14118/) (2023)
- [Physics Informed Token Transformer for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.08757/) (2023)
- [Polynomial Implicit Neural Representations For Large Diverse Datasets](https://mlsanity.com/paper/2303.11424/) (2023)
- [PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers](https://mlsanity.com/paper/2309.16816/) (2023)

## Built on by (most cited first, 30 of 35)

- [Recent Advances on Machine Learning for Computational Fluid Dynamics: A Survey](https://mlsanity.com/paper/2408.12171/) (2024)
- [Physics vs Distributions: Pareto Optimal Flow Matching with Physics Constraints](https://mlsanity.com/paper/2506.08604/) (2025)
- [Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries](https://mlsanity.com/paper/2602.04940/) (2026)
- [PhySense: Sensor Placement Optimization for Accurate Physics Sensing](https://mlsanity.com/paper/2505.18190/) (2025)
- [GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training](https://mlsanity.com/paper/2602.20399/) (2026)
- [Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanism](https://mlsanity.com/paper/2509.06322/) (2025)
- [Latent Neural Operator Pretraining for Solving Time-Dependent PDEs](https://mlsanity.com/paper/2410.20100/) (2024)
- [MATEY: multiscale adaptive foundation models for spatiotemporal physical systems](https://mlsanity.com/paper/2412.20601/) (2024)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [SuperWing: a comprehensive transonic wing dataset for data-driven aerodynamic design](https://mlsanity.com/paper/2512.14397/) (2025)
- [Latent Generative Solvers for Generalizable Long-Term Physics Simulation](https://mlsanity.com/paper/2602.11229/) (2026)
- [NESTOR: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2602.22059/) (2026)
- [Efficient Generative Transformer Operators For Million-Point PDEs](https://mlsanity.com/paper/2512.04974/) (2025)
- [Convolutional-neural-operator-based transfer learning for solving PDEs](https://mlsanity.com/paper/2512.17969/) (2025)
- [MVNN: A Measure-Valued Neural Network for Learning McKean-Vlasov Dynamics from Particle Data](https://mlsanity.com/paper/2604.00333/) (2026)
- [Harnessing AI for Inverse Partial Differential Equation Problems: Past, Present, and Prospects](https://mlsanity.com/paper/2605.16966/) (2026)
- [Neural operator discovery from heterogeneous trajectories](https://mlsanity.com/paper/2607.23337/) (2026)
- [Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs](https://mlsanity.com/paper/2510.05278/) (2025)
- [Brep2Shape: Boundary and Shape Representation Alignment via Self-Supervised Transformers](https://mlsanity.com/paper/2602.07429/) (2026)
- [Towards a Foundation-Model Paradigm for Aerodynamic Prediction in Three-dimensional Design](https://mlsanity.com/paper/2604.18062/) (2026)
- [Posterior-First Neural PDE Simulation: Inferring Hidden Problem State from a Single Field](https://mlsanity.com/paper/2605.03247/) (2026)
- [Replay-Based Continual Learning for Physics-Informed Neural Operators](https://mlsanity.com/paper/2605.04832/) (2026)
- [CATO: Charted Attention for Neural PDE Operators](https://mlsanity.com/paper/2605.09016/) (2026)
- [Large language models for partial differential equation workflows](https://mlsanity.com/paper/2608.03600/) (2026)
- [MoNo: Multiscale Optimal Transport Neural Operator for Solving PDEs on General Geometries](https://mlsanity.com/paper/2608.09764/) (2026)
- [Distillation of Foundation Models for Time-dependent PDEs](https://mlsanity.com/paper/2608.11937/) (2026)
- [STCO: Conditional Neural Operators for Time-Dependent PDEs](https://mlsanity.com/paper/2608.20477/) (2026)
- [Learnable composition for neural operators](https://mlsanity.com/paper/2609.03069/) (2026)
- [Enhancing Transformer Representations of Symbolic ODE Expressions](https://mlsanity.com/paper/2609.24746/) (2026)
- [PhysFieldBench: Can Multimodal Models Understand Physical Fields?](https://mlsanity.com/paper/2609.34072/) (2026)

## Abstract (arXiv)

Deep models have recently emerged as promising tools to solve partial differential equations (PDEs), known as neural PDE solvers. While neural solvers trained from either simulation data or physics-informed loss can solve PDEs reasonably well, they are mainly restricted to a few instances of PDEs, e.g. a certain equation with a limited set of coefficients. This limits their generalization to diverse PDEs, preventing them from being practical surrogate models of numerical solvers. In this paper, we present Unisolver, a novel Transformer model trained on diverse data and conditioned on diverse PDEs, aiming towards a universal neural PDE solver capable of solving a wide scope of PDEs. Instead of purely scaling up data and parameters, Unisolver stems from the theoretical analysis of the PDE-solving process. Inspired by the mathematical structure of PDEs that a PDE solution is fundamentally governed by a series of PDE components such as equation symbols and boundary conditions, we define a complete set of PDE components and flexibly embed them as domain-wise and point-wise deep conditions for Transformer PDE solvers. Integrating physical insights with recent Transformer advances, Unisolver achieves consistent state-of-the-art on three challenging large-scale benchmarks, showing impressive performance and generalizability. Code is available at https://github.com/thuml/Unisolver.