mlsanityData to 14 Oct 2026

Constant Memory Attention Block

arXiv 2306.12599 ↗First posted 21 Jun 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern foundation model architectures rely on attention mechanisms to effectively capture context. However, these methods require linear or quadratic memory in terms of the number of inputs/datapoints, limiting their applicability in low-compute domains. In this work, we propose Constant Memory Attention Block (CMAB), a novel general-purpose attention block that computes its output in constant memory and performs updates in constant computation. Highlighting CMABs efficacy, we introduce methods for Neural Processes and Temporal Point Processes. Empirically, we show our proposed methods achieve results competitive with state-of-the-art while being significantly more memory efficient.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 18 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

17 of 18 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  5. 2021Transformers in Vision: A Surveycited by 1,013
  6. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  7. 2018Conditional Neural Processescited by 426
  8. 2018Neural Processescited by 311
  9. 2019Attentive Neural Processescited by 283
  10. 2020Transformer Hawkes Processcited by 235
  11. 2019Intensity-Free Learning of Temporal Point Processescited by 140
  12. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 132
5 more
  1. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  2. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 85
  3. 2020Bootstrapping Neural Processescited by 40
  4. 2022Latent Bottlenecked Attentive Neural Processescited by 26
  5. 2023Meta Temporal Point Processescited by 17

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126