mlsanityData to 7 Oct 2026

Associative Recurrent Memory Transformer

arXiv 2407.04841 ↗First posted 5 Jul 2024cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

22 of 27 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
10 more
  1. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  6. 2014Memory Networkscited by 354
  7. 2022Memorizing Transformerscited by 219
  8. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  9. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  10. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  2. 2024Titans: Learning to Memorize at Test Timecited by 303
  3. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  4. 2025Mechanistic evaluation of Transformers and state space modelscited by 8
  5. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  6. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 6
  7. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoningcited by 5
  8. 2025Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Trainingcited by 3
  9. 2024Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAGcited by 2
  10. 2025Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformers for Long Contextscited by 2
  11. 2024Instructional Text Across Disciplines: A Survey of Representations, Downstream Tasks, and Open Challenges Toward Capable AI Agentscited by 1
  12. 2025Re:Frame -- Retrieving Experience From Associative Memorycited by 1
3 more
  1. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  2. 2025GatedFWA: Linear Flash Windowed Attention with Gated Associative Memorycited by 0
  3. 2026Beyond Episodic AI: Cognitive Field Networks for Biologically Inspired Persistent Cognitioncited by 0