mlsanityData to 7 Oct 2026

Translation Equivariant Transformer Neural Processes

arXiv 2406.12409 ↗First posted 18 Jun 2024stat.ML cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The effectiveness of neural processes (NPs) in modelling posterior prediction maps -- the mapping from data to posterior predictive distributions -- has significantly improved since their inception. This improvement can be attributed to two principal factors: (1) advancements in the architecture of permutation invariant set functions, which are intrinsic to all NPs; and (2) leveraging symmetries present in the true posterior predictive map, which are problem dependent. Transformers are a notable development in permutation invariant set functions, and their utility within NPs has been demonstrated through the family of models we refer to as TNPs. Despite significant interest in TNPs, little attention has been given to incorporating symmetries. Notably, the posterior prediction maps for data that are stationary -- a common assumption in spatio-temporal modelling -- exhibit translation equivariance. In this paper, we introduce of a new family of translation equivariant TNPs that incorporate translation equivariance. Through an extensive range of experiments on synthetic and real-world spatio-temporal data, we demonstrate the effectiveness of TE-TNPs relative to their non-translation-equivariant counterparts and other NP baselines.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 38 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

32 of 38 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  8. 2021E(n) Equivariant Graph Neural Networkscited by 951
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  11. 2021Machine learning accelerated computational fluid dynamicscited by 461
  12. 2018Conditional Neural Processescited by 421
20 more
  1. 2021Transformers Can Do Bayesian Inferencecited by 338
  2. 2018Neural Processescited by 307
  3. 2019Attentive Neural Processescited by 280
  4. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  5. 2019Convolutional Conditional Neural Processescited by 139
  6. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  7. 2020LieTransformer: Equivariant self-attention for Lie Groupscited by 100
  8. 2023Score-based Data Assimilationcited by 91
  9. 2021Learning with invariances in random features and kernel modelscited by 89
  10. 2020Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processescited by 74
  11. 2021Universal Approximation of Functions on Setscited by 62
  12. 2021The Gaussian Neural Processcited by 34
  13. 2020Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processescited by 29
  14. 2022Latent Bottlenecked Attentive Neural Processescited by 26
  15. 2022Practical Conditional Neural Processes Via Tractable Dependent Predictionscited by 25
  16. 2022The Neural Process Family: Survey, Applications and Perspectivescited by 23
  17. 2023A Neural PDE Solver with Temporal Stencil Modelingcited by 22
  18. 2022Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecastingcited by 17
  19. 2023Practical Equivariances via Relational Conditional Neural Processescited by 13
  20. 2023Exploring the Space of Key-Value-Query Models with Intentioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 4.

12 papers.

  1. 2025Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learningcited by 11
  2. 2024Approximately Equivariant Neural Processescited by 8
  3. 2025ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisitioncited by 8
  4. 2025Flow Matching Neural Processescited by 6
  5. 2025Dimension Agnostic Neural Processescited by 5
  6. 2024Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Datacited by 4
  7. 2025Sparse Gaussian Neural Processescited by 2
  8. 2025Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processescited by 1
  9. 2026Amortising Inference and Meta-Learning Priors in Neural Networkscited by 0
  10. 2026PRIM: Meta-Learned Bayesian Root Cause Analysiscited by 0
  11. 2026Spectral Transformer Neural Processescited by 0
  12. 2026Revisiting Neural Processes via Fourier Transform and Volterra Seriescited by 0