# Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning

arXiv: https://arxiv.org/abs/2506.10973 · HTML: https://mlsanity.com/paper/2506.10973/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 97 of its 129 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 97)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [Physics-Informed Neural Operator for Learning Partial Differential Equations](https://mlsanity.com/paper/2111.03794/) (2021)
- [Fourier Neural Operator with Learned Deformations for PDEs on General Geometries](https://mlsanity.com/paper/2207.05209/) (2022)
- [Message Passing Neural PDE Solvers](https://mlsanity.com/paper/2202.03376/) (2022)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)

## Built on by (most cited first, 12 of 12)

- [Decoupled Diffusion Sampling for Inverse Problems on Function Spaces](https://mlsanity.com/paper/2601.23280/) (2026)
- [Resolution-Agnostic Neural Operators for Multi-Rate Sparse-View CT](https://mlsanity.com/paper/2512.12236/) (2025)
- [Extending Neural Operators: Robust Handling of Functions Beyond the Training Set](https://mlsanity.com/paper/2603.03621/) (2026)
- [Learning Neural Operator Surrogates for the Black Hole Accretion Code](https://mlsanity.com/paper/2604.25985/) (2026)
- [Limits of Resolution Equivariance in Fourier Neural Operators](https://mlsanity.com/paper/2606.00677/) (2026)
- [Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs](https://mlsanity.com/paper/2606.28122/) (2026)
- [Orca: Neural Operators for Causal Reasoning in Continuous Time](https://mlsanity.com/paper/2607.27867/) (2026)
- [Neural Operator Learning for Collision-Aware Trajectory Planning of Spacecraft Swarms](https://mlsanity.com/paper/2608.00320/) (2026)
- [A neural operator view on U-Nets for inverse imaging problems](https://mlsanity.com/paper/2608.05839/) (2026)
- [Refinement Symmetry in Multimodal Transformers](https://mlsanity.com/paper/2609.32669/) (2026)
- [GeoWind2Plan: Mission-Time 3D Urban Wind Prediction for Energy-Efficient UAV Planning](https://mlsanity.com/paper/2609.36056/) (2026)
- [Counterfactual Predictions in Scientific Emulators Without Controlled Experiments](https://mlsanity.com/paper/2610.02252/) (2026)

## Abstract (arXiv)

A wide range of scientific problems, such as those described by continuous-time dynamical systems and partial differential equations (PDEs), are naturally formulated on function spaces. While function spaces are typically infinite-dimensional, deep learning has predominantly advanced through applications in computer vision and natural language processing that focus on mappings between finite-dimensional spaces. Such fundamental disparities in the nature of the data have limited neural networks from achieving a comparable level of success in scientific applications as seen in other fields. Neural operators are a principled way to generalize neural networks to mappings between function spaces, offering a pathway to replicate deep learning's transformative impact on scientific problems. For instance, neural operators can learn solution operators for entire classes of PDEs, e.g., physical systems with different boundary conditions, coefficient functions, and geometries. A key factor in deep learning's success has been the careful engineering of neural architectures through extensive empirical testing. Translating these neural architectures into neural operators allows operator learning to enjoy these same empirical optimizations. However, prior neural operator architectures have often been introduced as standalone models, not directly derived as extensions of existing neural network architectures. In this paper, we identify and distill the key principles for constructing practical implementations of mappings between infinite-dimensional function spaces. Using these principles, we propose a recipe for converting several popular neural architectures into neural operators with minimal modifications. This paper aims to guide practitioners through this process and details the steps to make neural operators work in practice. Our code can be found at https://github.com/neuraloperator/NNs-to-NOs