# In-Context In-Context Learning with Transformer Neural Processes

arXiv: https://arxiv.org/abs/2406.13493 · HTML: https://mlsanity.com/paper/2406.13493/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [Learning to Induce Causal Structure](https://mlsanity.com/paper/2204.04875/) (2022)
- [Universal Approximation of Functions on Sets](https://mlsanity.com/paper/2107.01959/) (2021)
- [Latent Bottlenecked Attentive Neural Processes](https://mlsanity.com/paper/2211.08458/) (2022)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [Episodic Multi-Task Learning with Heterogeneous Neural Processes](https://mlsanity.com/paper/2310.18713/) (2023)
- [Multi-Task Neural Processes](https://mlsanity.com/paper/2111.05820/) (2021)

## Built on by (most cited first, 4 of 4)

- [Amortized Probabilistic Conditioning for Optimization, Simulation and Inference](https://mlsanity.com/paper/2410.15320/) (2024)
- [Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens](https://mlsanity.com/paper/2411.05189/) (2024)
- [Sparse Gaussian Neural Processes](https://mlsanity.com/paper/2504.01650/) (2025)
- [Amortising Inference and Meta-Learning Priors in Neural Networks](https://mlsanity.com/paper/2602.08782/) (2026)

## Abstract (arXiv)

Neural processes (NPs) are a powerful family of meta-learning models that seek to approximate the posterior predictive map of the ground-truth stochastic process from which each dataset in a meta-dataset is sampled. There are many cases in which practitioners, besides having access to the dataset of interest, may also have access to other datasets that share similarities with it. In this case, integrating these datasets into the NP can improve predictions. We equip NPs with this functionality and describe this paradigm as in-context in-context learning. Standard NP architectures, such as the convolutional conditional NP (ConvCNP) or the family of transformer neural processes (TNPs), are not capable of in-context in-context learning, as they are only able to condition on a single dataset. We address this shortcoming by developing the in-context in-context learning pseudo-token TNP (ICICL-TNP). The ICICL-TNP builds on the family of PT-TNPs, which utilise pseudo-token-based transformer architectures to sidestep the quadratic computational complexity associated with regular transformer architectures. Importantly, the ICICL-TNP is capable of conditioning on both sets of datapoints and sets of datasets, enabling it to perform in-context in-context learning. We demonstrate the importance of in-context in-context learning and the effectiveness of the ICICL-TNP in a number of experiments.