# DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning

arXiv: https://arxiv.org/abs/2411.07239 · HTML: https://mlsanity.com/paper/2411.07239/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 56 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [Multiple Physics Pretraining for Physical Surrogate Models](https://mlsanity.com/paper/2310.02994/) (2023)
- [Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation](https://mlsanity.com/paper/2404.12355/) (2024)
- [PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics](https://mlsanity.com/paper/2409.09811/) (2024)
- [One-shot learning for solution operators of partial differential equations](https://mlsanity.com/paper/2104.05512/) (2021)
- [Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks](https://mlsanity.com/paper/2402.15406/) (2024)
- [A Foundation Model for the Earth System](https://mlsanity.com/paper/2405.13063/) (2024)
- [PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers](https://mlsanity.com/paper/2309.16816/) (2023)
- [DIMON: Learning Solution Operators of Partial Differential Equations on a Diffeomorphic Family of Domains](https://mlsanity.com/paper/2402.07250/) (2024)
- [LeMON: Learning to Learn Multi-Operator Networks](https://mlsanity.com/paper/2408.16168/) (2024)
- [On Approximating the Dynamic Response of Synchronous Generators via Operator Learning: A Step Towards Building Deep Operator-based Power Grid Simulators](https://mlsanity.com/paper/2301.12538/) (2023)
- [Efficient and generalizable nested Fourier-DeepONet for three-dimensional geological carbon sequestration](https://mlsanity.com/paper/2409.16572/) (2024)
- [Federated scientific machine learning for approximating functions and solving differential equations with data heterogeneity](https://mlsanity.com/paper/2410.13141/) (2024)
- [Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model](https://mlsanity.com/paper/2409.11609/) (2024)
- [Conformalized Prediction of Post-Fault Voltage Trajectories Using Pre-trained and Finetuned Attention-Driven Neural Operators](https://mlsanity.com/paper/2410.24162/) (2024)
- [Fine-Tune Language Models as Multi-Modal Differential Equation Solvers](https://mlsanity.com/paper/2308.05061/) (2023)
- [Disk2Planet: A Robust and Automated Machine Learning Tool for Parameter Inference in Disk-Planet Systems](https://mlsanity.com/paper/2409.17228/) (2024)

## Built on by (most cited first, 11 of 11)

- [VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction](https://mlsanity.com/paper/2411.16063/) (2024)
- [BCAT: A Block Causal Transformer for PDE Foundation Models for Fluid Dynamics](https://mlsanity.com/paper/2501.18972/) (2025)
- [PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations](https://mlsanity.com/paper/2512.23056/) (2025)
- [Learning PDE Solvers with Physics and Data: A Unifying View of Physics-Informed Neural Networks and Neural Operators](https://mlsanity.com/paper/2601.14517/) (2026)
- [Cauchy Random Features for Operator Learning in Sobolev Space](https://mlsanity.com/paper/2503.00300/) (2025)
- [Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models](https://mlsanity.com/paper/2505.10930/) (2025)
- [What do physics-informed DeepONets learn? Understanding and improving training for scientific computing applications](https://mlsanity.com/paper/2411.18459/) (2024)
- [Chain of Operators: An Inference-Time Harness for In-Context Operator Learning](https://mlsanity.com/paper/2606.12318/) (2026)
- [Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Data](https://mlsanity.com/paper/2601.13608/) (2026)
- [The Error of Deep Operator Networks Is the Sum of Its Parts: Branch-Trunk and Mode Error Decompositions](https://mlsanity.com/paper/2602.21910/) (2026)
- [PGD-NO: A Neural Operator with Precomputed Geometry Decomposition for 3D Million-scale Physics Simulations](https://mlsanity.com/paper/2607.08025/) (2026)

## Abstract (arXiv)

We propose a novel fine-tuning method to achieve multi-operator learning through training a distributed neural operator with diverse function data and then zero-shot fine-tuning the neural network using physics-informed losses for downstream tasks. Operator learning effectively approximates solution operators for PDEs and various PDE-related problems, yet it often struggles to generalize to new tasks. To address this, we investigate fine-tuning a pretrained model, while carefully selecting an initialization that enables rapid adaptation to new tasks with minimal data. Our approach combines distributed learning to integrate data from various operators in pre-training, while physics-informed methods enable zero-shot fine-tuning, minimizing the reliance on downstream data. We investigate standard fine-tuning and Low-Rank Adaptation fine-tuning, applying both to train complex nonlinear target operators that are difficult to learn only using random initialization. Through comprehensive numerical examples, we demonstrate the advantages of our approach, showcasing significant improvements in accuracy. Our findings provide a robust framework for advancing multi-operator learning and highlight the potential of transfer learning techniques in this domain.