mlsanityData to 7 Oct 2026

Titans: Learning to Memorize at Test Time

arXiv 2501.00663 ↗First posted 31 Dec 2024cs.LG cs.AI cs.CLcited by 303

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long past information. We show that this neural memory has the advantage of fast parallelizable training while maintaining a fast inference. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.

Key facts. Cited by 303 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 107 of its 135 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

107 of 135 references.

References · 107

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Mistral 7Bcited by 3,065
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  13. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  14. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  15. 2020Rethinking Attention with Performerscited by 1,584
  16. 2019Root Mean Square Layer Normalizationcited by 1,425
  17. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  18. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  19. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  20. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  21. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  22. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  23. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  24. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  25. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  26. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  27. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  28. 2014Neural Turing Machinescited by 630
  29. 2016Learning to learn by gradient descent by gradient descentcited by 616
  30. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  31. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  32. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  33. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  34. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  35. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  36. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  37. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  38. 2014Memory Networkscited by 354

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 108; 2026: 195.

303 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025Memory in the Age of AI Agentscited by 285
  3. 2026Reinforcement Learning via Self-Distillationcited by 285
  4. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  6. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  7. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025MemOS: A Memory OS for AI Systemcited by 129
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  11. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  12. 2025Test-Time Training Done Rightcited by 121
38 more
  1. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  2. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  3. 2025A Survey on Latent Reasoningcited by 71
  4. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  5. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  6. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  7. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  8. 2025Long-Context State-Space Video World Modelscited by 59
  9. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  10. 2025End-to-End Test-Time Training for Long Contextcited by 53
  11. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
  12. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 44
  13. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  14. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  15. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  16. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  17. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  18. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  19. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  20. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  21. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  22. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  23. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacitycited by 27
  24. 2026Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstructioncited by 25
  25. 2026ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Trainingcited by 23
  26. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 21
  27. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  28. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 19
  29. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 19
  30. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  31. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  32. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  33. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 16
  34. 2025TempoPFN: Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecastingcited by 16
  35. 2026Forest Before Trees: Latent Superposition for Efficient Visual Reasoningcited by 16
  36. 2025ViT3: Unlocking Test-Time Training in Visioncited by 15
  37. 2026In-Place Test-Time Trainingcited by 15
  38. 2026VGG-T3: Offline Feed-Forward 3D Reconstruction at Scalecited by 14

50 most cited of 303 All 303 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
  2. 2026PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory
  3. 2025Overflow Prevention Enhances Long-Context Recurrent LLMs
  4. 2025Revisiting associative recall in modern recurrent models
  5. 2024Stuffed Mamba: Oversized States Lead to the Inability to Forget
  6. 2025MemMamba: Rethinking Memory Patterns in State Space Model
  7. 2026Unlocking the Working Memory of Large Language Models for Latent Reasoning
  8. 2026On-Demand Attention: Language Models Know When to Recall