mlsanityData to 7 Oct 2026

TransformerFAM: Feedback attention is working memory

arXiv 2404.09173 ↗First posted 14 Apr 2024cs.LG cs.AI cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 67 of its 86 references that are in the map. First posted on arXiv 14 Apr 2024.

Builds on

67 of 86 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  14. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  15. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  16. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  17. 2020Reformer: The Efficient Transformercited by 1,658
  18. 2020Rethinking Attention with Performerscited by 1,584
  19. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  20. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  21. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  22. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  23. 2020On Layer Normalization in the Transformer Architecturecited by 906
  24. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  25. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  26. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  27. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  28. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  29. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  30. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  31. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  32. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  33. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  34. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  35. 2023Adapting Language Models to Compress Contextscited by 350
  36. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  37. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  38. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 7.

21 papers.

Sort:

By area:

  1. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  2. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  3. 2025Cognitive Memory in Large Language Modelscited by 28
  4. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  5. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  6. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modelingcited by 12
  7. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  8. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processingcited by 7
  9. 2024Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Trainingcited by 6
  10. 2026The Recurrent Transformer: Greater Effective Depth and Efficient Decodingcited by 5
  11. 2026Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behaviorcited by 5
  12. 2024CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modelingcited by 4
9 more
  1. 2024An Evolved Universal Transformer Memorycited by 4
  2. 2025Learning to Ponder: Adaptive Reasoning in Latent Spacecited by 4
  3. 2026Maglev: Sliding Recurrent Memorycited by 3
  4. 2026Prefix Sliding for efficient test-time scalingcited by 2
  5. 2025Attention is All You Need Until You Need Retentioncited by 1
  6. 2025Echo State Transformer: Attention Over Finite Memoriescited by 0
  7. 2026Sessa: Selective State Space Attentioncited by 0
  8. 2026TRAM: Enhancing Multimodal Reasoning with Trajectory-Derived Auxiliary Memorycited by 0
  9. 2026TraceRelay: Attention-Aligned Recurrence over Rolling Tracescited by 0