# Incremental Transformer Neural Processes

arXiv: https://arxiv.org/abs/2602.18955 · HTML: https://mlsanity.com/paper/2602.18955/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 21 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [Convergence of Sparse Variational Inference in Gaussian Processes Regression](https://mlsanity.com/paper/2008.00323/) (2020)
- [Streaming Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1705.07131/) (2017)
- [Autoregressive Conditional Neural Processes](https://mlsanity.com/paper/2303.14468/) (2023)
- [Latent Bottlenecked Attentive Neural Processes](https://mlsanity.com/paper/2211.08458/) (2022)
- [Kernel Interpolation for Scalable Online Gaussian Processes](https://mlsanity.com/paper/2103.01454/) (2021)
- [Environmental Sensor Placement with Convolutional Gaussian Neural Processes](https://mlsanity.com/paper/2211.10381/) (2022)
- [Memory Efficient Neural Processes via Constant Memory Attention Block](https://mlsanity.com/paper/2305.14567/) (2023)
- [Deep Stochastic Processes via Functional Markov Transition Operators](https://mlsanity.com/paper/2305.15574/) (2023)
- [Scalable Gaussian Processes for Characterizing Multidimensional Change Surfaces](https://mlsanity.com/paper/1511.04408/) (2015)
- [Efficient Autoregressive Inference for Transformer Probabilistic Models](https://mlsanity.com/paper/2510.09477/) (2025)

## Built on by (most cited first, 2 of 2)

- [Gaussian Mixture Copula Processes for Irregular Time Series](https://mlsanity.com/paper/2605.23632/) (2026)
- [NPBoost: Neural Processes with Gradient-Boosted Fixed Effects](https://mlsanity.com/paper/2609.28122/) (2026)

## Abstract (arXiv)

Neural Processes (NPs), and specifically Transformer Neural Processes (TNPs), have demonstrated remarkable performance across tasks ranging from spatiotemporal forecasting to tabular data modelling. However, many of these applications are inherently sequential, involving continuous data streams such as real-time sensor readings or database updates. In such settings, models should support cheap, incremental updates rather than recomputing internal representations from scratch for every new observation -- a capability existing TNP variants lack. Drawing inspiration from Large Language Models, we introduce the Incremental TNP (incTNP). By leveraging causal masking, Key-Value (KV) caching, and a data-efficient autoregressive training strategy, incTNP matches the predictive performance of standard TNPs while reducing the computational cost of updates from quadratic to linear time complexity. We empirically evaluate our model on a range of synthetic and real-world tasks, including tabular regression and temperature prediction. Our results show that, surprisingly, incTNP delivers performance comparable to -- or better than -- non-causal TNPs while unlocking orders-of-magnitude speedups for sequential inference. Finally, we assess the consistency of the model's updates -- by adapting a metric of "implicit Bayesianness", we show that under a one-at-a-time streaming protocol, incTNP retains a prediction rule as implicitly Bayesian as standard non-causal TNPs, demonstrating that incTNP achieves the computational benefits of causal masking without sacrificing the consistency required for streaming inference.