# Translation Equivariant Transformer Neural Processes

arXiv: https://arxiv.org/abs/2406.12409 · HTML: https://mlsanity.com/paper/2406.12409/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [LieTransformer: Equivariant self-attention for Lie Groups](https://mlsanity.com/paper/2012.10885/) (2020)
- [Score-based Data Assimilation](https://mlsanity.com/paper/2306.10574/) (2023)
- [Learning with invariances in random features and kernel models](https://mlsanity.com/paper/2102.13219/) (2021)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [Universal Approximation of Functions on Sets](https://mlsanity.com/paper/2107.01959/) (2021)
- [The Gaussian Neural Process](https://mlsanity.com/paper/2101.03606/) (2021)
- [Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes](https://mlsanity.com/paper/2011.12916/) (2020)
- [Latent Bottlenecked Attentive Neural Processes](https://mlsanity.com/paper/2211.08458/) (2022)
- [Practical Conditional Neural Processes Via Tractable Dependent Predictions](https://mlsanity.com/paper/2203.08775/) (2022)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [A Neural PDE Solver with Temporal Stencil Modeling](https://mlsanity.com/paper/2302.08105/) (2023)
- [Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting](https://mlsanity.com/paper/2206.09450/) (2022)

## Built on by (most cited first, 13 of 13)

- [Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learning](https://mlsanity.com/paper/2507.05526/) (2025)
- [Approximately Equivariant Neural Processes](https://mlsanity.com/paper/2406.13488/) (2024)
- [ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition](https://mlsanity.com/paper/2506.07259/) (2025)
- [Flow Matching Neural Processes](https://mlsanity.com/paper/2512.23853/) (2025)
- [Dimension Agnostic Neural Processes](https://mlsanity.com/paper/2502.20661/) (2025)
- [Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Data](https://mlsanity.com/paper/2410.06731/) (2024)
- [Sparse Gaussian Neural Processes](https://mlsanity.com/paper/2504.01650/) (2025)
- [Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processes](https://mlsanity.com/paper/2506.09163/) (2025)
- [Amortising Inference and Meta-Learning Priors in Neural Networks](https://mlsanity.com/paper/2602.08782/) (2026)
- [PRIM: Meta-Learned Bayesian Root Cause Analysis](https://mlsanity.com/paper/2605.08786/) (2026)
- [Spectral Transformer Neural Processes](https://mlsanity.com/paper/2605.09498/) (2026)
- [Revisiting Neural Processes via Fourier Transform and Volterra Series](https://mlsanity.com/paper/2606.01172/) (2026)
- [Amortized Counterfactual Prediction with Graph Neural Networks](https://mlsanity.com/paper/2610.13724/) (2026)

## Abstract (arXiv)

The effectiveness of neural processes (NPs) in modelling posterior prediction maps -- the mapping from data to posterior predictive distributions -- has significantly improved since their inception. This improvement can be attributed to two principal factors: (1) advancements in the architecture of permutation invariant set functions, which are intrinsic to all NPs; and (2) leveraging symmetries present in the true posterior predictive map, which are problem dependent. Transformers are a notable development in permutation invariant set functions, and their utility within NPs has been demonstrated through the family of models we refer to as TNPs. Despite significant interest in TNPs, little attention has been given to incorporating symmetries. Notably, the posterior prediction maps for data that are stationary -- a common assumption in spatio-temporal modelling -- exhibit translation equivariance. In this paper, we introduce of a new family of translation equivariant TNPs that incorporate translation equivariance. Through an extensive range of experiments on synthetic and real-world spatio-temporal data, we demonstrate the effectiveness of TE-TNPs relative to their non-translation-equivariant counterparts and other NP baselines.