mlsanityData to 7 Oct 2026

UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation

arXiv 2403.07187 ↗First posted 11 Mar 2024cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

We present Unified PDE Solvers (UPS), a data- and compute-efficient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which is data-demanding and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

32 of 45 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2020Neural Operator: Graph Kernel Network for Partial Differential Equationscited by 722
  10. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  11. 2021Machine learning accelerated computational fluid dynamicscited by 461
  12. 2022PDEBENCH: An Extensive Benchmark for Scientific Machine Learningcited by 396
20 more
  1. 2019Axial Attention in Multidimensional Transformerscited by 350
  2. 2021Choose a Transformer: Fourier or Galerkincited by 333
  3. 2023GNOT: A General Neural Operator Transformer for Operator Learningcited by 321
  4. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  5. 2022Transformer for Partial Differential Equations' Operator Learningcited by 270
  6. 2021Factorized Fourier Neural Operatorscited by 237
  7. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  8. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  9. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  10. 2023Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behaviorcited by 128
  11. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  12. 2019Learning Neural PDE Solvers with Convergence Guaranteescited by 77
  13. 2022Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equationscited by 55
  14. 2023CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamicscited by 40
  15. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  16. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  17. 2023Deep Equilibrium Based Neural Operators for Steady-State PDEscited by 19
  18. 2022Efficient Architecture Search for Diverse Taskscited by 18
  19. 2023Reprogramming Pretrained Language Models for Protein Sequence Representation Learningcited by 12
  20. 2023Gene Set Summarization using Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 11.

27 papers.

Sort:

By area:

  1. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
  2. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  3. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  4. 2024Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Spacecited by 10
  5. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  6. 2025Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanismcited by 7
  7. 2026AutoNumerics: An Autonomous, PDE-Agnostic Multi-Agent Pipeline for Scientific Computingcited by 7
  8. 2024Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Textcited by 6
  9. 2025PI-MFM: Physics-informed multimodal foundation model for solving partial differential equationscited by 6
  10. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  11. 2026Structure-Preserving Learning Improves Geometry Generalization in Neural PDEscited by 5
  12. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 5
15 more
  1. 2024Data-Efficient Inference of Neural Fluid Fields via SciML Foundation Modelcited by 4
  2. 2026CompNO: A Novel Foundation Model approach for solving Partial Differential Equationscited by 4
  3. 2026OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inferencecited by 4
  4. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  5. 2025PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passescited by 3
  6. 2025RECODE: Reasoning Through Code Generation for Visual Question Answeringcited by 2
  7. 2025Neuro-Symbolic AI for Analytical Solutions of Differential Equationscited by 1
  8. 2025SPUS: A Lightweight and Parameter-Efficient Foundation Model for PDEscited by 0
  9. 2025FMint-SDE: A Multimodal Foundation Model for Accelerating Numerical Simulation of SDEs via Error Correctioncited by 0
  10. 2026Compositional Neural Operators for Multi-Dimensional Fluid Dynamicscited by 0
  11. 2026Generalized Neural Operator for Parametric and Boundary-Value Problemscited by 0
  12. 2026Hybrid Lagrangian-Eulerian Model for Lagrangian Fluid Simulationcited by 0
  13. 2026Large language models for partial differential equation workflowscited by 0
  14. 2026Distillation of Foundation Models for Time-dependent PDEscited by 0
  15. 2026DiffPDE: Masked Diffusion Language Models as PDE Solvercited by 0