mlsanityData to 7 Oct 2026

Memory Mosaics

arXiv 2405.06394 ↗First posted 10 May 2024cs.LG cs.AI cs.NEcited by 16

Not on the map. Why?

Abstract (arXiv)

Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way ("predictive disentanglement"). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2017Dynamic Routing Between Capsulescited by 689
  10. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2020Hopfield Networks is All You Needcited by 461
13 more
  1. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  2. 2014Memory Networkscited by 354
  3. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  4. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  5. 2019Augmenting Self-attention with Persistent Memorycited by 92
  6. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  7. 2013Deep Learning of Representations: Looking Forwardcited by 84
  8. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  9. 2023Energy Transformercited by 69
  10. 2019SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systemscited by 46
  11. 2022Disentanglement of Correlated Factors via Hausdorff Factorized Supportcited by 37
  12. 2018Disentangling the independently controllable factors of variation by interacting with the worldcited by 16
  13. 2017A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 8.

16 papers.

Sort:

By area:

  1. 2024Titans: Learning to Memorize at Test Timecited by 303
  2. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  3. 2024Strong Model Collapsecited by 62
  4. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  5. 2026Fast-weight Product Key Memorycited by 11
  6. 2025Composer: A Search Framework for Hybrid Neural Architecture Designcited by 7
  7. 2025Memory Mosaics at scalecited by 5
  8. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  9. 2024Learning with Hidden Factorial Structurecited by 2
  10. 2026Sparse Attention as Compact Kernel Regressioncited by 2
  11. 2026Online Vector Quantized Attentioncited by 1
  12. 2024Rethinking Associative Memory Mechanism in Induction Headcited by 0
4 more
  1. 2026Sinkhorn Based Associative Memory Retrieval Using Spherical Hellinger Kantorovich Dynamicscited by 0
  2. 2026Mela: Test-Time Memory Consolidation based on Transformation Hypothesiscited by 0
  3. 2026Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracycited by 0
  4. 2026Memory as a cache: Exact context reuse and deletion by constructioncited by 0