# A foundational neural operator that continuously learns without forgetting

arXiv: https://arxiv.org/abs/2310.18885 · HTML: https://mlsanity.com/paper/2310.18885/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data](https://mlsanity.com/paper/1901.06314/) (2019)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks](https://mlsanity.com/paper/1906.05747/) (2019)
- [Transfer learning enhanced physics informed neural network for phase-field modeling of fracture](https://mlsanity.com/paper/1907.02531/) (2019)
- [Transfer learning based multi-fidelity physics informed deep neural network](https://mlsanity.com/paper/2005.10614/) (2020)
- [Operator-valued Kernels for Learning from Functional Response Data](https://mlsanity.com/paper/1510.08231/) (2015)
- [On the Power of Foundation Models](https://mlsanity.com/paper/2211.16327/) (2022)
- [Physics informed WNO](https://mlsanity.com/paper/2302.05925/) (2023)
- [Gated Linear Networks](https://mlsanity.com/paper/1910.01526/) (2019)
- [BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning](https://mlsanity.com/paper/2305.04769/) (2023)
- [Koopman operator for time-dependent reliability analysis](https://mlsanity.com/paper/2203.02658/) (2022)
- [A Combinatorial Perspective on Transfer Learning](https://mlsanity.com/paper/2010.12268/) (2020)

## Built on by (most cited first, 11 of 11)

- [Masked Autoencoders are PDE Learners](https://mlsanity.com/paper/2403.17728/) (2024)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [Generative adversarial wavelet neural operator: Application to fault detection and isolation of multivariate time series data](https://mlsanity.com/paper/2401.04004/) (2024)
- [Pretraining a Neural Operator in Lower Dimensions](https://mlsanity.com/paper/2407.17616/) (2024)
- [Neural Operator induced Gaussian Process framework for probabilistic solution of parametric partial differential equations](https://mlsanity.com/paper/2404.15618/) (2024)
- [Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator](https://mlsanity.com/paper/2405.06910/) (2024)
- [Harnessing physics-informed operators for high-dimensional reliability analysis problems](https://mlsanity.com/paper/2409.04708/) (2024)
- [From Local Interactions to Global Operators: Scalable Gaussian Process Operator for Physical Systems](https://mlsanity.com/paper/2506.15906/) (2025)
- [Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs](https://mlsanity.com/paper/2507.03863/) (2025)
- [SPLICE: Sparse Physics Library Identification and Compositional Expansion](https://mlsanity.com/paper/2610.17029/) (2026)

## Abstract (arXiv)

Machine learning has witnessed substantial growth, leading to the development of advanced artificial intelligence models crafted to address a wide range of real-world challenges spanning various domains, such as computer vision, natural language processing, and scientific computing. Nevertheless, the creation of custom models for each new task remains a resource-intensive undertaking, demanding considerable computational time and memory resources. In this study, we introduce the concept of the Neural Combinatorial Wavelet Neural Operator (NCWNO) as a foundational model for scientific computing. This model is specifically designed to excel in learning from a diverse spectrum of physics and continuously adapt to the solution operators associated with parametric partial differential equations (PDEs). The NCWNO leverages a gated structure that employs local wavelet experts to acquire shared features across multiple physical systems, complemented by a memory-based ensembling approach among these local wavelet experts. This combination enables rapid adaptation to new challenges. The proposed foundational model offers two key advantages: (i) it can simultaneously learn solution operators for multiple parametric PDEs, and (ii) it can swiftly generalize to new parametric PDEs with minimal fine-tuning. The proposed NCWNO is the first foundational operator learning algorithm distinguished by its (i) robustness against catastrophic forgetting, (ii) the maintenance of positive transfer for new parametric PDEs, and (iii) the facilitation of knowledge transfer across dissimilar tasks. Through an extensive set of benchmark examples, we demonstrate that the NCWNO can outperform task-specific baseline operator learning frameworks with minimal hyperparameter tuning at the prediction stage. We also show that with minimal fine-tuning, the NCWNO performs accurate combinatorial learning of new parametric PDEs.