mlsanityData to 7 Oct 2026

Memory Efficient Neural Processes via Constant Memory Attention Block

arXiv 2305.14567 ↗First posted 23 May 2023cs.LG cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural Processes (NPs) are popular meta-learning methods for efficiently modelling predictive uncertainty. Recent state-of-the-art methods, however, leverage expensive attention mechanisms, limiting their applications, particularly in low-resource settings. In this work, we propose Constant Memory Attentive Neural Processes (CMANPs), an NP variant that only requires constant memory. To do so, we first propose an efficient update operation for Cross Attention. Leveraging the update operation, we propose Constant Memory Attention Block (CMAB), a novel attention block that (i) is permutation invariant, (ii) computes its output in constant memory, and (iii) performs constant computation updates. Finally, building on CMAB, we detail Constant Memory Attentive Neural Processes. Empirically, we show CMANPs achieve state-of-the-art results on popular NP benchmarks while being significantly more memory efficient than prior methods.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

39 of 46 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2017Deep Setscited by 1,394
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  7. 2021Transformers in Vision: A Surveycited by 1,009
  8. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  11. 2019Predicting Dynamic Embedding Trajectory in Temporal Interaction Networkscited by 443
  12. 2018Conditional Neural Processescited by 421
27 more
  1. 2021A Survey of Transformerscited by 399
  2. 2018Neural Processescited by 307
  3. 2019Attentive Neural Processescited by 280
  4. 2020Transformer Hawkes Processcited by 230
  5. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  6. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  7. 2019Convolutional Conditional Neural Processescited by 139
  8. 2019Intensity-Free Learning of Temporal Point Processescited by 138
  9. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  10. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  11. 2022Transformers as Meta-Learners for Implicit Neural Representationscited by 79
  12. 2020Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processescited by 74
  13. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  14. 2019Sequential Neural Processescited by 55
  15. 2020Bootstrapping Neural Processescited by 40
  16. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 36
  17. 2021Task-adaptive Neural Process for User Cold-Start Recommendationcited by 32
  18. 2023Autoregressive Conditional Neural Processescited by 32
  19. 2020Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processescited by 29
  20. 2021Convolutional conditional neural processes for local climate downscalingcited by 29
  21. 2022Latent Bottlenecked Attentive Neural Processescited by 26
  22. 2022The Neural Process Family: Survey, Applications and Perspectivescited by 23
  23. 2023Meta Temporal Point Processescited by 16
  24. 2019Recurrent Neural Processescited by 10
  25. 2022Meta-Learning Regrasping Strategies for Physical-Agnostic Objectscited by 7
  26. 2021How Neural Processes Improve Graph Link Predictioncited by 6
  27. 2022Category-Agnostic 6D Pose Estimation with Conditional Neural Processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 1.

9 papers.

  1. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  2. 2024Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasauruscited by 8
  3. 2025ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisitioncited by 8
  4. 2024Attention as an RNNcited by 6
  5. 2025Dimension Agnostic Neural Processescited by 5
  6. 2024Gridded Transformer Neural Processes for Large Unstructured Spatio-Temporal Datacited by 4
  7. 2025Efficient Autoregressive Inference for Transformer Probabilistic Modelscited by 4
  8. 2026Incremental Transformer Neural Processescited by 2
  9. 2025Scalable Spatiotemporal Inference with Biased Scan Attention Transformer Neural Processescited by 1