# In-Context Operator Learning with Data Prompts for Differential Equation Problems

arXiv: https://arxiv.org/abs/2304.07993 · HTML: https://mlsanity.com/paper/2304.07993/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations](https://mlsanity.com/paper/1706.04702/) (2017)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [Learning Operators with Coupled Attention](https://mlsanity.com/paper/2201.01032/) (2022)
- [Deep transfer operator learning for partial differential equations under conditional shift](https://mlsanity.com/paper/2204.09810/) (2022)
- [Transfer learning based multi-fidelity physics informed deep neural network](https://mlsanity.com/paper/2005.10614/) (2020)
- [Reliable extrapolation of deep neural operators informed by physics or sparse observations](https://mlsanity.com/paper/2212.06347/) (2022)
- [Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarios](https://mlsanity.com/paper/2205.07731/) (2022)
- [Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domains](https://mlsanity.com/paper/2104.10873/) (2021)
- [Learning Functional Priors and Posteriors from Data and Physics](https://mlsanity.com/paper/2106.05863/) (2021)
- [Meta-Auto-Decoder for Solving Parametric Partial Differential Equations](https://mlsanity.com/paper/2111.08823/) (2021)
- [Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games](https://mlsanity.com/paper/2002.10113/) (2020)
- [Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations](https://mlsanity.com/paper/2212.04663/) (2022)
- [Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow](https://mlsanity.com/paper/2206.03198/) (2022)

## Built on by (most cited first, 30 of 101)

- [DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2403.03542/) (2024)
- [The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning](https://mlsanity.com/paper/2412.00568/) (2024)
- [Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation](https://mlsanity.com/paper/2404.12355/) (2024)
- [Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solvers](https://mlsanity.com/paper/2405.17527/) (2024)
- [PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics](https://mlsanity.com/paper/2409.09811/) (2024)
- [Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning](https://mlsanity.com/paper/2402.15734/) (2024)
- [PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations](https://mlsanity.com/paper/2505.24717/) (2025)
- [PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws](https://mlsanity.com/paper/2401.07364/) (2024)
- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction](https://mlsanity.com/paper/2411.16063/) (2024)
- [DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction](https://mlsanity.com/paper/2504.19496/) (2025)
- [FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Model](https://mlsanity.com/paper/2404.14688/) (2024)
- [BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics](https://mlsanity.com/paper/2501.18972/) (2025)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solving](https://mlsanity.com/paper/2408.02198/) (2024)
- [PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers](https://mlsanity.com/paper/2309.16816/) (2023)
- [Zebra: In-Context Generative Pretraining for Solving Parametric PDEs](https://mlsanity.com/paper/2410.03437/) (2024)
- [Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation](https://mlsanity.com/paper/2502.00782/) (2025)
- [Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods](https://mlsanity.com/paper/2506.10959/) (2025)
- [Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods](https://mlsanity.com/paper/2311.11798/) (2023)
- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [MODNO: Multi Operator Learning With Distributed Neural Operators](https://mlsanity.com/paper/2404.02892/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [LeMON: Learning to Learn Multi-Operator Networks](https://mlsanity.com/paper/2408.16168/) (2024)
- [MORPH: PDE Foundation Models with Arbitrary Data Modality](https://mlsanity.com/paper/2509.21670/) (2025)
- [Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training](https://mlsanity.com/paper/2510.25803/) (2025)
- [AgenticSciML: Collaborative Multi-Agent Systems for Emergent Discovery in Scientific Machine Learning](https://mlsanity.com/paper/2511.07262/) (2025)
- [A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptions](https://mlsanity.com/paper/2502.06026/) (2025)
- [Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equations](https://mlsanity.com/paper/2509.05186/) (2025)
- [Test-time Generalization for Physics through Neural Operator Splitting](https://mlsanity.com/paper/2602.00884/) (2026)

## Abstract (arXiv)

This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON), to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, we can not only get rid of retraining (even fine-tuning) the neural network for new problems, but also leverage the commonalities shared across operators so that only a few demos in the prompt are needed when learning a new operator. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations (ODEs), partial differential equations (PDEs), and mean-field control (MFC) problems, and also show that it can generalize its learning capability to operators beyond the training distribution.