# Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network

arXiv: https://arxiv.org/abs/2201.02217 · HTML: https://mlsanity.com/paper/2201.02217/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 6 Jan 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems](https://mlsanity.com/paper/1710.00211/) (2017)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Solving ill-posed inverse problems using iterative deep neural networks](https://mlsanity.com/paper/1704.04058/) (2017)
- [Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks](https://mlsanity.com/paper/1905.13166/) (2019)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Continuous Graph Neural Networks](https://mlsanity.com/paper/1912.00967/) (2019)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability](https://mlsanity.com/paper/1906.03663/) (2019)
- [EikoNet: Solving the Eikonal equation with Deep Neural Networks](https://mlsanity.com/paper/2004.00361/) (2020)
- [Graph Element Networks: adaptive, structured computation and memory](https://mlsanity.com/paper/1904.09019/) (2019)
- [Non-Local Color Image Denoising with Convolutional Neural Networks](https://mlsanity.com/paper/1611.06757/) (2016)
- [Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems](https://mlsanity.com/paper/1904.05417/) (2019)
- [Numerical methods for nonlocal and fractional models](https://mlsanity.com/paper/2002.01401/) (2020)
- [Nonlocal Neural Networks, Nonlocal Diffusion and Nonlocal Modeling](https://mlsanity.com/paper/1806.00681/) (2018)

## Built on by (most cited first, 30 of 45)

- [MIONet: Learning multiple-input operators via tensor product](https://mlsanity.com/paper/2202.06137/) (2022)
- [Physics-Informed Deep Neural Operator Networks](https://mlsanity.com/paper/2207.05748/) (2022)
- [Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport](https://mlsanity.com/paper/2204.06684/) (2022)
- [Learning Deep Implicit Fourier Neural Operators (IFNOs) with Applications to Heterogeneous Material Modeling](https://mlsanity.com/paper/2203.08205/) (2022)
- [Reliable extrapolation of deep neural operators informed by physics or sparse observations](https://mlsanity.com/paper/2212.06347/) (2022)
- [Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems](https://mlsanity.com/paper/2204.09157/) (2022)
- [Interfacing Finite Elements with Deep Neural Operators for Fast Multiscale Modeling of Mechanics Problems](https://mlsanity.com/paper/2203.00003/) (2022)
- [Domain Agnostic Fourier Neural Operators](https://mlsanity.com/paper/2305.00478/) (2023)
- [SVD Perspectives for Augmenting DeepONet Flexibility and Interpretability](https://mlsanity.com/paper/2204.12670/) (2022)
- [Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems](https://mlsanity.com/paper/2212.03130/) (2022)
- [INO: Invariant Neural Operators for Learning Complex Physical Systems with Momentum Conservation](https://mlsanity.com/paper/2212.14365/) (2022)
- [Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing](https://mlsanity.com/paper/2211.07377/) (2022)
- [Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws](https://mlsanity.com/paper/2312.11176/) (2023)
- [PIXEL: Physics-Informed Cell Representations for Fast and Accurate PDE Solvers](https://mlsanity.com/paper/2207.12800/) (2022)
- [A Physics-Guided Neural Operator Learning Approach to Model Biological Tissues from Digital Image Correlation Measurements](https://mlsanity.com/paper/2204.00205/) (2022)
- [Learning in latent spaces improves the predictive accuracy of deep neural operators](https://mlsanity.com/paper/2304.07599/) (2023)
- [Peridynamic Neural Operators: A Data-Driven Nonlocal Constitutive Model for Complex Material Responses](https://mlsanity.com/paper/2401.06070/) (2024)
- [Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery](https://mlsanity.com/paper/2408.07307/) (2024)
- [Harnessing Scale and Physics: A Multi-Graph Neural Operator Framework for PDEs on Arbitrary Geometries](https://mlsanity.com/paper/2411.15178/) (2024)
- [IAE-Net: Integral Autoencoders for Discretization-Invariant Learning](https://mlsanity.com/paper/2203.05142/) (2022)
- [Deep Neural Operator Enabled Digital Twin Modeling for Additive Manufacturing](https://mlsanity.com/paper/2405.09572/) (2024)
- [Multi-fidelity wavelet neural operator with application to uncertainty quantification](https://mlsanity.com/paper/2208.05606/) (2022)
- [Deep-OSG: Deep Learning of Operators in Semigroup](https://mlsanity.com/paper/2302.03358/) (2023)
- [GNN-based physics solver for time-independent PDEs](https://mlsanity.com/paper/2303.15681/) (2023)
- [Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology](https://mlsanity.com/paper/2310.05227/) (2023)
- [Nonparametric learning of kernels in nonlocal operators](https://mlsanity.com/paper/2205.11006/) (2022)
- [MetaNO: How to Transfer Your Knowledge on Learning Hidden Physics](https://mlsanity.com/paper/2301.12095/) (2023)
- [Adaptive physics-informed neural operator for coarse-grained non-equilibrium flows](https://mlsanity.com/paper/2210.15799/) (2022)
- [A physics-informed transformer neural operator for learning generalized solutions of initial boundary value problems](https://mlsanity.com/paper/2412.09009/) (2024)
- [Integration of physics-informed operator learning and finite element method for parametric learning of partial differential equations](https://mlsanity.com/paper/2401.02363/) (2024)

## Abstract (arXiv)

Neural operators have recently become popular tools for designing solution maps between function spaces in the form of neural networks. Differently from classical scientific machine learning approaches that learn parameters of a known partial differential equation (PDE) for a single instance of the input parameters at a fixed resolution, neural operators approximate the solution map of a family of PDEs. Despite their success, the uses of neural operators are so far restricted to relatively shallow neural networks and confined to learning hidden governing laws. In this work, we propose a novel nonlocal neural operator, which we refer to as nonlocal kernel network (NKN), that is resolution independent, characterized by deep neural networks, and capable of handling a variety of tasks such as learning governing equations and classifying images. Our NKN stems from the interpretation of the neural network as a discrete nonlocal diffusion reaction equation that, in the limit of infinite layers, is equivalent to a parabolic nonlocal equation, whose stability is analyzed via nonlocal vector calculus. The resemblance with integral forms of neural operators allows NKNs to capture long-range dependencies in the feature space, while the continuous treatment of node-to-node interactions makes NKNs resolution independent. The resemblance with neural ODEs, reinterpreted in a nonlocal sense, and the stable network dynamics between layers allow for generalization of NKN's optimal parameters from shallow to deep networks. This fact enables the use of shallow-to-deep initialization techniques. Our tests show that NKNs outperform baseline methods in both learning governing equations and image classification tasks and generalize well to different resolutions and depths.