mlsanityData to 7 Oct 2026

In-Context Operator Learning with Data Prompts for Differential Equation Problems

arXiv 2304.07993 ↗First posted 17 Apr 2023cs.LG cs.NA math.NAcited by 98

Not on the map. Why?

Abstract (arXiv)

This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON), to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, we can not only get rid of retraining (even fine-tuning) the neural network for new problems, but also leverage the commonalities shared across operators so that only a few demos in the prompt are needed when learning a new operator. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations (ODEs), partial differential equations (PDEs), and mean-field control (MFC) problems, and also show that it can generalize its learning capability to operators beyond the training distribution.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

32 of 45 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  10. 2017The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problemscited by 581
  11. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  12. 2021Learning the solution operator of parametric partial differential equations with physics-informed DeepOnetscited by 537
20 more
  1. 2021Machine learning accelerated computational fluid dynamicscited by 461
  2. 2017PDE-Net: Learning PDEs from Datacited by 304
  3. 2020Model Reduction and Neural Networks for Parametric PDEscited by 257
  4. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  5. 2017Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equationscited by 188
  6. 2019Transfer learning enhanced physics informed neural network for phase-field modeling of fracturecited by 116
  7. 2019A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problemscited by 95
  8. 2022Learning Operators with Coupled Attentioncited by 88
  9. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  10. 2020Transfer learning based multi-fidelity physics informed deep neural networkcited by 68
  11. 2022Reliable extrapolation of deep neural operators informed by physics or sparse observationscited by 62
  12. 2022Transfer learning based physics-informed neural networks for solving inverse problems in engineering structures under different loading scenarioscited by 44
  13. 2021Mosaic Flows: A Transferable Deep Learning Framework for Solving PDEs on Unseen Domainscited by 37
  14. 2021Learning Functional Priors and Posteriors from Data and Physicscited by 37
  15. 2021Meta-Auto-Decoder for Solving Parametric Partial Differential Equationscited by 36
  16. 2020Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Gamescited by 35
  17. 2022Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equationscited by 28
  18. 2022Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flowcited by 20
  19. 2020Analysis of three dimensional potential problems in non-homogeneous media with physics-informed deep collocation method using material transfer learning and sensitivity analysiscited by 11
  20. 2023Multi-fidelity prediction of fluid flow and temperature field based on transfer learning using Fourier Neural Operatorcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 26; 2025: 39; 2026: 29.

98 papers.

Sort:

By area:

  1. 2024DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Trainingcited by 137
  2. 2024The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learningcited by 134
  3. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 45
  4. 2024PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamicscited by 33
  5. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
  6. 2024Unisolver: PDE-Conditional Transformers Towards Universal Neural PDE Solverscited by 31
  7. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 30
  8. 2025PDE-Transformer: Efficient and Versatile Transformers for Physics Simulationscited by 29
  9. 2024Zero-shot forecasting of chaotic systemscited by 25
  10. 2024VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Predictioncited by 22
  11. 2025DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic predictioncited by 21
  12. 2024FMint: Bridging Human Designed and Data Pretrained Models for Differential Equation Foundation Modelcited by 18
38 more
  1. 2024Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solvingcited by 17
  2. 2023PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformerscited by 16
  3. 2025BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamicscited by 16
  4. 2023Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methodscited by 14
  5. 2024MODNO: Multi Operator Learning With Distributed Neural Operatorscited by 14
  6. 2024Strategies for Pretraining Neural Operatorscited by 14
  7. 2024Zebra: In-Context Generative Pretraining for Solving Parametric PDEscited by 14
  8. 2025Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptationcited by 14
  9. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  10. 2024LeMON: Learning to Learn Multi-Operator Networkscited by 13
  11. 2025Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methodscited by 13
  12. 2025AgenticSciML: Collaborative Multi-Agent Systems for Emergent Discovery in Scientific Machine Learningcited by 13
  13. 2025MORPH: PDE Foundation Models with Arbitrary Data Modalitycited by 12
  14. 2024Masked Autoencoders are PDE Learnerscited by 11
  15. 2025Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Trainingcited by 10
  16. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  17. 2025Probabilistic operator learning: generative modeling and uncertainty quantification for foundation models of differential equationscited by 9
  18. 2024OmniArch: Building Foundation Model For Scientific Computingcited by 8
  19. 2024BiLO: Bilevel Local Operator Learning for PDE Inverse Problemscited by 8
  20. 2025A Multimodal PDE Foundation Model for Prediction and Scientific Text Descriptionscited by 8
  21. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  22. 2025The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operatorscited by 8
  23. 2026Test-time Generalization for Physics through Neural Operator Splittingcited by 8
  24. 2026GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Trainingcited by 8
  25. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  26. 2024Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Modelcited by 6
  27. 2025A Deep Learning Framework for Multi-Operator Learning: Architectures and Approximation Theorycited by 6
  28. 2025PI-MFM: Physics-informed multimodal foundation model for solving partial differential equationscited by 6
  29. 2024Pretraining a Neural Operator in Lower Dimensionscited by 5
  30. 2024In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learningcited by 5
  31. 2025Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Modelscited by 5
  32. 2025Solving Optimal Execution Problems via In-Context Operator Networkscited by 4
  33. 2025Neural Implicit Solution Formula for Efficiently Solving Hamilton-Jacobi Equationscited by 4
  34. 2025FD-Bench: A Modular and Fair Benchmark for Data-driven Fluid Simulationcited by 4
  35. 2026CompNO: A Novel Foundation Model approach for solving Partial Differential Equationscited by 4
  36. 2023Fine-Tune Language Models as Multi-Modal Differential Equation Solverscited by 3
  37. 2024Unsupervised Solution Operator Learning for Mean-Field Games via Sampling-Invariant Parametrizationscited by 3
  38. 2024A Misleading Gallery of Fluid Motion by Generative Artificial Intelligencecited by 3

50 most cited of 98 All 98 (JSON)