mlsanityData to 14 Oct 2026

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

arXiv 2407.04620 ↗First posted 5 Jul 2024cs.LG cs.AI cs.CLcited by 355

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.

Key facts. Cited by 355 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 53 of its 88 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

53 of 88 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  4. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
37 more
  1. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  4. 2016Learning to learn by gradient descent by gradient descentcited by 620
  5. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  7. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  8. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  9. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  10. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  11. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 286
  12. 2020Consistent Video Depth Estimationcited by 220
  13. 2022Test-Time Training with Masked Autoencoderscited by 212
  14. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  15. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  16. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  17. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  18. 2013Learning by Transductioncited by 121
  19. 2017A Tutorial on Kernel Density Estimation and Recent Advancescited by 101
  20. 2022The Devil in Linear Transformercited by 98
  21. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  22. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  23. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  24. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  25. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  26. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  27. 2020Meta Learning Backpropagation And Improving Itcited by 49
  28. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 46
  29. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  30. 2018Meta-Learning Update Rules for Unsupervised Representation Learningcited by 37
  31. 2023Test-Time Training on Video Streamscited by 37
  32. 2019Dynamic Evaluation of Transformer Language Modelscited by 23
  33. 2022Meta-Learning Fast Weight Language Modelscited by 21
  34. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  35. 2023Learning to (Learn at Test Time)cited by 17
  36. 2022Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rulescited by 16
  37. 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 40; 2025: 122; 2026: 193.

355 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  2. 2026Reinforcement Learning via Self-Distillationcited by 332
  3. 2024Titans: Learning to Memorize at Test Timecited by 328
  4. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  6. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  7. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  9. 2025Test-Time Training Done Rightcited by 130
  10. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 129
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
38 more
  1. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  2. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  3. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  4. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  5. 2025A Survey on Latent Reasoningcited by 81
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 76
  7. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  8. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  9. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  10. 2025Transformers without Normalizationcited by 70
  11. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  12. 2026Attention Residualscited by 65
  13. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 64
  14. 2025Long-Context State-Space Video World Modelscited by 64
  15. 2025End-to-End Test-Time Training for Long Contextcited by 62
  16. 2024Synthetic continued pretrainingcited by 60
  17. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 59
  18. 2024Beyond Model Adaptation at Test Time: A Surveycited by 59
  19. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 55
  20. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  21. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  22. 2025MoM: Linear Sequence Modeling with Mixture-of-Memoriescited by 47
  23. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 47
  24. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 43
  25. 2025Centaur: Robust End-to-End Autonomous Driving with Test-Time Trainingcited by 42
  26. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  27. 2025Log-Linear Attentioncited by 40
  28. 2026LoGeR: Long-Context Geometric Reconstruction with Hybrid Memorycited by 39
  29. 2024LION: Linear Group RNN for 3D Object Detection in Point Cloudscited by 38
  30. 2025Human3R: Everyone Everywhere All at Oncecited by 37
  31. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  32. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 33
  33. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 31
  34. 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
  35. 2026Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstructioncited by 31
  36. 2026ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Trainingcited by 26
  37. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  38. 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 24

50 most cited of 355 All 355 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
  2. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
  3. 2026Expressivity-Efficiency Tradeoffs for Hybrid Sequence Models
  4. 2025Element-wise Attention Is All You Need
  5. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks
  6. 2025Measuring In-Context Computation Complexity via Hidden State Prediction
  7. 2024Attention as an RNN
  8. 2024The Hidden Attention of Mamba Models