# Neural Processes with Stochastic Attention: Paying more attention to the context dataset

arXiv: https://arxiv.org/abs/2204.05449 · HTML: https://mlsanity.com/paper/2204.05449/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [Meta-Learning Requires Meta-Augmentation](https://mlsanity.com/paper/2007.05549/) (2020)
- [Sequential Neural Processes](https://mlsanity.com/paper/1906.10264/) (2019)
- [Bayesian Attention Modules](https://mlsanity.com/paper/2010.10604/) (2020)
- [Bootstrapping Neural Processes](https://mlsanity.com/paper/2008.02956/) (2020)
- [Latent Alignment and Variational Attention](https://mlsanity.com/paper/1807.03756/) (2018)
- [MetaFun: Meta-Learning with Iterative Functional Updates](https://mlsanity.com/paper/1912.02738/) (2019)
- [WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling](https://mlsanity.com/paper/1803.01328/) (2018)
- [Variational Attention for Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.08207/) (2017)
- [Robustifying Sequential Neural Processes](https://mlsanity.com/paper/2006.15987/) (2020)
- [Meta-Learning surrogate models for sequential decision making](https://mlsanity.com/paper/1903.11907/) (2019)
- [GLocal-K: Global and Local Kernels for Recommender Systems](https://mlsanity.com/paper/2108.12184/) (2021)
- [Learning Hard Alignments with Variational Inference](https://mlsanity.com/paper/1705.05524/) (2017)

## Built on by (most cited first, 16 of 16)

- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion](https://mlsanity.com/paper/2304.08183/) (2023)
- [Graph Neural Processes for Spatio-Temporal Extrapolation](https://mlsanity.com/paper/2305.18719/) (2023)
- [NPCL: Neural Processes for Uncertainty-Aware Continual Learning](https://mlsanity.com/paper/2310.19272/) (2023)
- [Versatile Neural Processes for Learning Implicit Neural Representations](https://mlsanity.com/paper/2301.08883/) (2023)
- [Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation](https://mlsanity.com/paper/2304.01518/) (2023)
- [Episodic Multi-Task Learning with Heterogeneous Neural Processes](https://mlsanity.com/paper/2310.18713/) (2023)
- [Bridge the Inference Gaps of Neural Processes via Expectation Maximization](https://mlsanity.com/paper/2501.03264/) (2025)
- [Spectral Convolutional Conditional Neural Processes](https://mlsanity.com/paper/2404.13182/) (2024)
- [Adaptive Conditional Quantile Neural Processes](https://mlsanity.com/paper/2305.18777/) (2023)
- [Real-time Adaptation for Condition Monitoring Signal Prediction using Label-aware Neural Processes](https://mlsanity.com/paper/2403.16377/) (2024)
- [Exploiting Inferential Structure in Neural Processes](https://mlsanity.com/paper/2306.15169/) (2023)
- [Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes](https://mlsanity.com/paper/2505.01726/) (2025)
- [Geometric Neural Process Fields](https://mlsanity.com/paper/2502.02338/) (2025)
- [Decoupling Distance and Networks: Hybrid Graph Attention-Geostatistical Methods for Spatio-temporal Risk Mapping](https://mlsanity.com/paper/2603.08393/) (2026)
- [Revisiting Neural Processes via Fourier Transform and Volterra Series](https://mlsanity.com/paper/2606.01172/) (2026)

## Abstract (arXiv)

Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. NPs essentially leverage a given dataset as a context representation to derive a suitable identifier for a novel task. To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant. In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information. From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing NPs to consider features in a target dataset and context embedding independently. We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings. We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion. Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.