# Neural Integral Equations

arXiv: https://arxiv.org/abs/2209.15190 · HTML: https://mlsanity.com/paper/2209.15190/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 89 references that are in the map.
- First posted on arXiv 30 Sep 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Learning two-phase microstructure evolution using neural operators and autoencoder architectures](https://mlsanity.com/paper/2204.07230/) (2022)
- [STEER: Simple Temporal Regularization For Neural ODEs](https://mlsanity.com/paper/2006.10711/) (2020)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021)
- [Neural Integro-Differential Equations](https://mlsanity.com/paper/2206.14282/) (2022)
- [Integral Equations and Machine Learning](https://mlsanity.com/paper/1712.06115/) (2017)

## Built on by (most cited first, 23 of 23)

- [Integrating Neural Operators with Diffusion Models Improves Spectral Representation in Turbulence Modeling](https://mlsanity.com/paper/2409.08477/) (2024)
- [Temporal Neural Operator for Modeling Time-Dependent Physical Phenomena](https://mlsanity.com/paper/2504.20249/) (2025)
- [The Volterra signature](https://mlsanity.com/paper/2603.04525/) (2026)
- [Real-Time Sensing of Inaccessible Physical Fields via an Edge-Deployable Hardware-Portable Graph Neural Operator](https://mlsanity.com/paper/2604.01802/) (2026)
- [CaLMFlow: Volterra Flow Matching using Causal Language Models](https://mlsanity.com/paper/2410.05292/) (2024)
- [Spectral methods for Neural Integral Equations](https://mlsanity.com/paper/2312.05654/) (2023)
- [Projection Methods for Operator Learning and Universal Approximation](https://mlsanity.com/paper/2406.12264/) (2024)
- [Universal Approximation of Operators with Transformers and Neural Integral Operators](https://mlsanity.com/paper/2409.00841/) (2024)
- [Physics-informed neural networks viewpoint for solving the Dyson-Schwinger equations of quantum electrodynamics](https://mlsanity.com/paper/2411.02177/) (2024)
- [A Neural Operator-Based Approach to Symbolic Discovery of PDEs](https://mlsanity.com/paper/2501.08086/) (2025)
- [Next Generation Equation-Free Multiscale Modelling of Crowd Dynamics via Machine Learning](https://mlsanity.com/paper/2508.03926/) (2025)
- [Leray-Schauder Mappings for Operator Learning](https://mlsanity.com/paper/2410.01746/) (2024)
- [Banach neural operator for Navier-Stokes equations](https://mlsanity.com/paper/2512.09070/) (2025)
- [Rates and architectures for learning geometrically non-trivial operators](https://mlsanity.com/paper/2512.09376/) (2025)
- [Neural Discovery of Memory and Nonlocal Kernels in Integro-Differential Equations with Constrained Kolmogorov--Arnold Networks](https://mlsanity.com/paper/2607.11110/) (2026)
- [Neural Integral Operators for Inverse Problems: An Operator-Learning Framework for Small-Sample Spectroscopic Classification](https://mlsanity.com/paper/2505.03677/) (2025)
- [Circuit-Inspired High-Order Neural Networks with Unified Neural Dynamics Modeling for PDE Solving and Visual Perception](https://mlsanity.com/paper/2603.23977/) (2026)
- [Learning Contractive Integral Operators with Fredholm Integral Neural Operators](https://mlsanity.com/paper/2604.03034/) (2026)
- [A Structure-Preserving Graph Neural Solver for Parametric Hyperbolic Conservation Laws](https://mlsanity.com/paper/2604.15617/) (2026)
- [Nonlocal operator learning for fMRI encoding and decoding tasks](https://mlsanity.com/paper/2605.20389/) (2026)
- [Starter-Iterator Neural Operator: A Unified Architecture for High-Fidelity Forward and Inverse PDE Problems](https://mlsanity.com/paper/2606.18305/) (2026)
- [MnemoDyn: Learning Resting State Dynamics from 40K FMRI sequences](https://mlsanity.com/paper/2608.23936/) (2026)
- [Neural Harmonic Measure Operator](https://mlsanity.com/paper/2609.35752/) (2026)

## Abstract (arXiv)

Nonlinear operators with long distance spatiotemporal dependencies are fundamental in modeling complex systems across sciences, yet learning these nonlocal operators remains challenging in machine learning. Integral equations (IEs), which model such nonlocal systems, have wide ranging applications in physics, chemistry, biology, and engineering. We introduce Neural Integral Equations (NIE), a method for learning unknown integral operators from data using an IE solver. To improve scalability and model capacity, we also present Attentional Neural Integral Equations (ANIE), which replaces the integral with self-attention. Both models are grounded in the theory of second kind integral equations, where the indeterminate appears both inside and outside the integral operator. We provide theoretical analysis showing how self-attention can approximate integral operators under mild regularity assumptions, further deepening previously reported connections between transformers and integration, and deriving corresponding approximation results for integral operators. Through numerical benchmarks on synthetic and real world data, including Lotka-Volterra, Navier-Stokes, and Burgers' equations, as well as brain dynamics and integral equations, we showcase the models' capabilities and their ability to derive interpretable dynamics embeddings. Our experiments demonstrate that ANIE outperforms existing methods, especially for longer time intervals and higher dimensional problems. Our work addresses a critical gap in machine learning for nonlocal operators and offers a powerful tool for studying unknown complex systems with long range dependencies.