# Explain Like I'm Five: Using LLMs to Improve PDE Surrogate Models with Text

arXiv: https://arxiv.org/abs/2410.01137 · HTML: https://mlsanity.com/paper/2410.01137/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Scalable Transformer for PDE Surrogate Modeling](https://mlsanity.com/paper/2305.17560/) (2023)
- [Query2CAD: Generating CAD models using natural language queries](https://mlsanity.com/paper/2406.00144/) (2024)
- [UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation](https://mlsanity.com/paper/2403.07187/) (2024)
- [Physics Informed Token Transformer for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.08757/) (2023)
- [FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model](https://mlsanity.com/paper/2404.14688/) (2024)
- [LLM-3D Print: Large Language Models To Monitor and Control 3D Printing](https://mlsanity.com/paper/2408.14307/) (2024)
- [PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers](https://mlsanity.com/paper/2309.16816/) (2023)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [MyCrunchGPT: A chatGPT assisted framework for scientific machine learning](https://mlsanity.com/paper/2306.15551/) (2023)
- [LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models](https://mlsanity.com/paper/2406.08648/) (2024)
- [Logically Constrained Robotics Transformers for Enhanced Perception-Action Planning](https://mlsanity.com/paper/2408.05336/) (2024)

## Built on by (most cited first, 7 of 7)

- [PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs](https://mlsanity.com/paper/2502.00963/) (2025)
- [PDEAgent-Bench: A Multi-Metric, Multi-Library Benchmark for PDE Solver Generation](https://mlsanity.com/paper/2605.09636/) (2026)
- [Text-Trained LLMs Can Zero-Shot Extrapolate PDE Dynamics, Revealing a Three-Stage In-Context Learning Mechanism](https://mlsanity.com/paper/2509.06322/) (2025)
- [AdditiveLLM: Large Language Models Predict Defects in Additive Manufacturing](https://mlsanity.com/paper/2501.17784/) (2025)
- [OpInf-LLM: Parametric PDE Solving with LLMs via Operator Inference](https://mlsanity.com/paper/2602.01493/) (2026)
- [Domain Adapted Large Language Models for Additive Manufacturing](https://mlsanity.com/paper/2603.22017/) (2026)
- [Does Text Steer Neural PDE Surrogates? A Controlled Diagnostic with OperatorCLIP](https://mlsanity.com/paper/2609.38517/) (2026)

## Abstract (arXiv)

Solving Partial Differential Equations (PDEs) is ubiquitous in science and engineering. Computational complexity and difficulty in writing numerical solvers has motivated the development of data-driven machine learning techniques to generate solutions quickly. The recent rise in popularity of Large Language Models (LLMs) has enabled easy integration of text in multimodal machine learning models, allowing easy integration of additional system information such as boundary conditions and governing equations through text. In this work, we explore using pretrained LLMs to integrate various amounts of known system information into PDE learning. Using FactFormer as our testing backbone, we add a multimodal block to fuse numerical and textual information. We compare sentence-level embeddings, word-level embeddings, and a standard tokenizer across 2D Heat, Burgers, Navier-Stokes, and Shallow-Water data sets. These challenging benchmarks show that pretrained LLMs are able to utilize text descriptions of system information and enable accurate prediction using only initial conditions.