# Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries

arXiv: https://arxiv.org/abs/2602.04940 · HTML: https://mlsanity.com/paper/2602.04940/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries](https://mlsanity.com/paper/2502.02414/) (2025)
- [Solving High-Dimensional PDEs with Latent Spectral Models](https://mlsanity.com/paper/2301.12664/) (2023)
- [Geometry Aware Operator Transformer as an Efficient and Accurate Neural Surrogate for PDEs on Arbitrary Domains](https://mlsanity.com/paper/2505.18781/) (2025)
- [Improved Operator Learning by Orthogonal Attention](https://mlsanity.com/paper/2310.12487/) (2023)
- [AB-UPT: Scaling Neural CFD Surrogates for High-Fidelity Automotive Aerodynamics Simulations via Anchored-Branched Universal Physics Transformers](https://mlsanity.com/paper/2502.09692/) (2025)
- [Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers](https://mlsanity.com/paper/2405.17527/) (2024)
- [DrivAerML: High-Fidelity Computational Fluid Dynamics Dataset for Road-Car External Aerodynamics](https://mlsanity.com/paper/2408.11969/) (2024)
- [RoPINN: Region Optimized Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.14369/) (2024)
- [AhmedML: High-Fidelity Computational Fluid Dynamics Dataset for Incompressible, Low-Speed Bluff Body Aerodynamics](https://mlsanity.com/paper/2407.20801/) (2024)
- [PhySense: Sensor Placement Optimization for Accurate Physics Sensing](https://mlsanity.com/paper/2505.18190/) (2025)

## Built on by (most cited first, 20 of 20)

- [GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training](https://mlsanity.com/paper/2602.20399/) (2026)
- [RETO: A Rotary-Enhanced Transformer Operator for High-Fidelity Prediction of Automotive Aerodynamics](https://mlsanity.com/paper/2605.00062/) (2026)
- [ShardTensor: Domain Parallelism for Scientific Machine Learning](https://mlsanity.com/paper/2605.11111/) (2026)
- [Adapting Automotive Aerodynamics Surrogates to New Vehicle Families via Transfer Learning](https://mlsanity.com/paper/2605.27968/) (2026)
- [Learning Neural Operator Surrogates for the Black Hole Accretion Code](https://mlsanity.com/paper/2604.25985/) (2026)
- [M3: Reframing Training Measures for Discretized Physical Simulations](https://mlsanity.com/paper/2605.08843/) (2026)
- [Neural Statistical Functions](https://mlsanity.com/paper/2605.11327/) (2026)
- [Courant: a State-Adaptive Perceiver-Based Neural Surrogate with Local Support and Interpretable Field Decomposition](https://mlsanity.com/paper/2605.25115/) (2026)
- [Let There Be Light: Reflection, Refraction and Scattering for Neural Operators](https://mlsanity.com/paper/2606.03262/) (2026)
- [Attention mechanism for scalable mesh-based neural surrogates of free-surface fluids](https://mlsanity.com/paper/2606.23251/) (2026)
- [Physics Transformer: Tailoring Transformer for General PDE Prediction](https://mlsanity.com/paper/2607.24513/) (2026)
- [FLARE++: Low-rank attention with dynamic attention routing](https://mlsanity.com/paper/2608.11519/) (2026)
- [HI-MeshGraphNets: Efficient and Accurate Mesh-based Physics Learning with Hierarchical Multi-scale Graph Neural Networks](https://mlsanity.com/paper/2608.13827/) (2026)
- [Distance-Aware Attention and Wall-Distance Expert Routing for Transformer-Based 3D Flow Prediction](https://mlsanity.com/paper/2609.07222/) (2026)
- [MENO: Memory-Efficient Neural Operator](https://mlsanity.com/paper/2609.27739/) (2026)
- [HGPTrans: Hierarchical Graph-Pooling Transolver for Automotive Aerodynamic Drag Coefficient Prediction](https://mlsanity.com/paper/2609.31765/) (2026)
- [Does Transolver really need a Transformer?](https://mlsanity.com/paper/2609.32525/) (2026)
- [Neural Harmonic Measure Operator](https://mlsanity.com/paper/2609.35752/) (2026)
- [Transolver-σ: Joint Spectral-Physical Subspace Modeling for Neural PDE Solving](https://mlsanity.com/paper/2609.37279/) (2026)
- [Cross-Domain Pretraining for Steady-State Neural CFD Surrogates](https://mlsanity.com/paper/2610.10398/) (2026)

## Abstract (arXiv)

Deep learning has emerged as a transformative tool for the neural surrogate modeling of partial differential equations (PDEs), known as neural PDE solvers. However, scaling these solvers to industrial-scale geometries with over 108 cells remains a fundamental challenge due to the prohibitive memory complexity of processing high-resolution meshes. We present Transolver-3, a new member of the Transolver family as a highly scalable framework designed for high-fidelity physics simulations. To bridge the gap between limited GPU capacity and the resolution requirements of complex engineering tasks, we introduce two key architectural optimizations: faster slice and deslice by exploiting matrix multiplication associative property and geometry slice tiling to partition the computation of physical states. Combined with an amortized training strategy by learning on random subsets of original high-resolution meshes and a physical state caching technique during inference, Transolver-3 enables high-fidelity field prediction on industrial-scale meshes. Extensive experiments demonstrate that Transolver-3 can handle meshes with over 160 million cells, achieving impressive performance across three challenging simulation benchmarks, including aircraft and automotive design tasks. Code is available at https://github.com/thuml/Transolver-3.