mlsanityData to 14 Oct 2026

∞-former: Infinite Memory Transformer

arXiv 2109.00301 ↗First posted 1 Sep 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the ∞-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the ∞-former's attention complexity becomes independent of the context length, trading off memory length with precision. In order to control where precision is more important, ∞-former maintains "sticky memories" being able to model arbitrarily long contexts while keeping the computation budget fixed. Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the ∞-former's ability to retain information from long sequences.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 1 Sep 2021.

Builds on

42 of 54 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2016Layer Normalizationcited by 5,669
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Reformer: The Efficient Transformercited by 1,668
30 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  4. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  7. 2020Efficient Transformers: A Surveycited by 790
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2014Neural Turing Machinescited by 644
  11. 2019Defending Against Neural Fake Newscited by 559
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  13. 2014Memory Networkscited by 363
  14. 2021Random Feature Attentioncited by 354
  15. 2020Sparse Sinkhorn Attentioncited by 251
  16. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  17. 2020Fast Transformers with Clustered Attentioncited by 123
  18. 2018A Dataset for Document Grounded Conversationscited by 104
  19. 2021Adaptive Semiparametric Language Modelscited by 83
  20. 2016Improving Neural Language Models with a Continuous Cachecited by 82
  21. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  22. 2015Learning to Transduce with Unbounded Memorycited by 54
  23. 2020Augmenting Transformers with KNN-Based Composite Memory for Dialoguecited by 53
  24. 2020Hard-Coded Gaussian Attention for Neural Machine Translationcited by 42
  25. 2020Sparse and Continuous Attention Mechanismscited by 39
  26. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  27. 2016Scaling Memory-Augmented Neural Networks with Sparse Reads and Writescited by 28
  28. 2020Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encodingcited by 19
  29. 2016Hierarchical Memory Networkscited by 15
  30. 2021Multimodal Continuous Visual Attention Mechanismscited by 6

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 0; 2025: 1.

7 papers.

  1. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  2. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  3. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  4. 2022Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencodercited by 15
  5. 2022MuLD: The Multitask Long Document Benchmarkcited by 9
  6. 2022LaMemo: Language Modeling with Look-Ahead Memorycited by 4
  7. 2025BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Modelscited by 1