# Attentive Neural Processes

arXiv: https://arxiv.org/abs/1901.05761 · HTML: https://mlsanity.com/paper/1901.05761/

## Key facts

- Cited by 283 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2019.

## Built on by (most cited first, 30 of 283)

- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision](https://mlsanity.com/paper/2306.11719/) (2023)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Generative Models as Distributions of Functions](https://mlsanity.com/paper/2102.04776/) (2021)
- [Conservative Objective Models for Effective Offline Model-Based Optimization](https://mlsanity.com/paper/2107.06882/) (2021)
- [Score-based Diffusion Models in Function Space](https://mlsanity.com/paper/2302.07400/) (2023)
- [MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images](https://mlsanity.com/paper/2106.11944/) (2021)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Continuous PDE Dynamics Forecasting with Implicit Neural Representations](https://mlsanity.com/paper/2209.14855/) (2022)
- [Learning Physical Models that Can Respect Conservation Laws](https://mlsanity.com/paper/2302.11002/) (2023)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [Generalizing to New Physical Systems via Context-Informed Dynamics Model](https://mlsanity.com/paper/2202.01889/) (2022)
- [Neural Diffusion Processes](https://mlsanity.com/paper/2206.03992/) (2022)
- [Diffusion Generative Models in Infinite Dimensions](https://mlsanity.com/paper/2212.00886/) (2022)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Modeling Temporal Data as Continuous Functions with Stochastic Process Diffusion](https://mlsanity.com/paper/2211.02590/) (2022)
- [Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation](https://mlsanity.com/paper/2102.07970/) (2021)
- [You Never Cluster Alone](https://mlsanity.com/paper/2106.01908/) (2021)
- [Few-Shot Diffusion Models](https://mlsanity.com/paper/2205.15463/) (2022)
- [Gradient-Free Generation for Hard-Constrained Systems](https://mlsanity.com/paper/2412.01786/) (2024)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Online Adaptation of Language Models with a Memory of Amortized Contexts](https://mlsanity.com/paper/2403.04317/) (2024)
- [Active Evaluation Acquisition for Efficient LLM Benchmarking](https://mlsanity.com/paper/2410.05952/) (2024)
- [The Gaussian Neural Process](https://mlsanity.com/paper/2101.03606/) (2021)
- [Context-Aware Safe Reinforcement Learning for Non-Stationary Environments](https://mlsanity.com/paper/2101.00531/) (2021)

## Abstract (arXiv)

Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.