mlsanityData to 14 Oct 2026

End-to-End Test-Time Training for Long Context

arXiv 2512.23675 ↗First posted 29 Dec 2025cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 72 of its 104 references that are in the map. First posted on arXiv 29 Dec 2025.

Builds on

72 of 104 references.

References · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
  1. 2016Learning without Forgettingcited by 2,496
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  11. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  13. 2016Learning to learn by gradient descent by gradient descentcited by 620
  14. 2019Three scenarios for continual learningcited by 614
  15. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  16. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  17. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  18. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  19. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  20. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  21. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  22. 2014Deeply-Supervised Netscited by 367
  23. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
  24. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  25. 2024Titans: Learning to Memorize at Test Timecited by 328
  26. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 286
  27. 2020Consistent Video Depth Estimationcited by 220
  28. 2022Test-Time Training with Masked Autoencoderscited by 212
  29. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  30. 2023Continual Pre-training of Language Modelscited by 198
  31. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  32. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  33. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  34. 2025Test-Time Training Done Rightcited by 130
  35. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  36. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  37. 2021An Attention Free Transformercited by 112
  38. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 62.

62 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  3. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  4. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  5. 2026RoboTTT: Context Scaling for Robot Policiescited by 19
  6. 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
  7. 2026Test-Time Training with KV Binding Is Secretly Linear Attentioncited by 13
  8. 2026Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Trainingcited by 11
  9. 2026GradMem: Learning to Write Context into Memory with Test-Time Gradient Descentcited by 9
  10. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  11. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  12. 2026When Does Continual Learning Require Learningcited by 7
38 more
  1. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  2. 2026Scaling Self-Evolving Agents via Parametric Memorycited by 5
  3. 2026Learning to Self-Evolvecited by 4
  4. 2026PACEvolve++: Improving Test-time Learning for Evolutionary Search Agentscited by 4
  5. 2026Memory for Large Language Modelscited by 4
  6. 2026Fast Spatial Memory with Elastic Test-Time Trainingcited by 3
  7. 2026AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agentscited by 3
  8. 2026Modular TTT: Rethinking Test-Time Training as Composable Modulescited by 3
  9. 2026A Survey on Self-Improving Test-Time Intelligence: Feedback-Driven Adapting, Learning, and Scaling at Inferencecited by 3
  10. 2026Test-Time Learning of Causal Structure from Interventional Datacited by 2
  11. 2026Test-Time Training with Next-Token Predictioncited by 2
  12. 2026Learning What to Remember: Test-Time Training via Context Distillationcited by 2
  13. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  14. 2026Prefix Sliding for efficient test-time scalingcited by 2
  15. 2026Can Video World Models Track Unobserved World States?cited by 2
  16. 2026Reinforced Fast Weights with Next-Sequence Predictioncited by 1
  17. 2026Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoningcited by 1
  18. 2026Multi-Mixer Models: Flexible Sequence Modeling with Shared Representationscited by 1
  19. 2026Latent Personal Memory: Represent personal memory as dynamic soft promptscited by 1
  20. 2026Self-Guided Test-Time Training for Long-Context LLMscited by 1
  21. 2026Rethinking Expressivity and Efficiency in Test-Time Trainingcited by 1
  22. 2026Agentic-TTT: Training test-time policy for test-time trainingcited by 1
  23. 2026Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberationcited by 0
  24. 2026Beyond Attention: True Adaptive World Models via Spherical Kernel Operatorcited by 0
  25. 2026MemDLM: Memory-Enhanced DLM Trainingcited by 0
  26. 2026FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuningcited by 0
  27. 2026In-context learning to predict critical transitions in dynamical systemscited by 0
  28. 2026Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencecited by 0
  29. 2026RW-TTT: Batched Serving for Request-Owned Test-Time Training Statecited by 0
  30. 2026T3R: Deeper Test-Time Adaptation for Graph Neural Networks via Gradient Rotationcited by 0
  31. 2026Memoir: Should a Model Write to Its Memory While It Thinks?cited by 0
  32. 2026Continual Learning in Transitioncited by 0
  33. 2026ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decouplingcited by 0
  34. 2026LeCor: Learning to Be Corrected by Meta-Learned Test-Time Training for Interactive 3D Lung-Tumour Segmentationcited by 0
  35. 2026FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attentioncited by 0
  36. 2026ProTTT: Learning to Learn Semantic User Memory with Test-Time Trainingcited by 0
  37. 2026DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregationcited by 0
  38. 2026GradLev: Token-Parallel Test-Time Training Via Costate Predictioncited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Role of Training Data in Test-Time Scaling
  2. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
  3. 2025Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
  4. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  5. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  6. 2026TEMPO: Scaling Test-time Training for Large Reasoning Models
  7. 2026Learning When to Attend: Conditional Memory Access for Long-Context LLMs
  8. 2025Overflow Prevention Enhances Long-Context Recurrent LLMs