# In-Place Test-Time Training

arXiv: https://arxiv.org/abs/2604.06169 · HTML: https://mlsanity.com/paper/2604.06169/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 61 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)

## Built on by (most cited first, 20 of 20)

- [When Does Continual Learning Require Learning](https://mlsanity.com/paper/2607.07847/) (2026)
- [Metis: Memory Foundation Model](https://mlsanity.com/paper/2607.26760/) (2026)
- [No Time Like the Present: Agentic Test-Time Training for LLM Agents](https://mlsanity.com/paper/2607.03441/) (2026)
- [Memory for Large Language Models](https://mlsanity.com/paper/2607.25380/) (2026)
- [AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?](https://mlsanity.com/paper/2608.00155/) (2026)
- [Absorber LLM: Harnessing Causal Synchronization for Test-Time Training](https://mlsanity.com/paper/2604.20915/) (2026)
- [Query-Conditioned Test-Time Self-Training for Large Language Models](https://mlsanity.com/paper/2605.13369/) (2026)
- [Test-Time Training with Next-Token Prediction](https://mlsanity.com/paper/2606.21803/) (2026)
- [Learning What to Remember: Test-Time Training via Context Distillation](https://mlsanity.com/paper/2608.01672/) (2026)
- [Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training](https://mlsanity.com/paper/2607.00368/) (2026)
- [Self-Guided Test-Time Training for Long-Context LLMs](https://mlsanity.com/paper/2607.09415/) (2026)
- [Atria Dawn: The Dawn of Agentic Superintelligence](https://mlsanity.com/paper/2609.15818/) (2026)
- [GradLev: Token-Parallel Test-Time Training Via Costate Prediction](https://mlsanity.com/paper/2609.34174/) (2026)
- [Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling](https://mlsanity.com/paper/2609.35109/) (2026)
- [Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation](https://mlsanity.com/paper/2610.05076/) (2026)
- [Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models](https://mlsanity.com/paper/2610.05282/) (2026)
- [Towards In-Parameter Memory Augmentation for Large Language Models](https://mlsanity.com/paper/2610.08630/) (2026)
- [From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning](https://mlsanity.com/paper/2610.11373/) (2026)
- [Recurrent Harness for Agentic Continual Learning](https://mlsanity.com/paper/2610.12668/) (2026)
- [Encoding Context into LLM Weights: What Makes It Context?](https://mlsanity.com/paper/2610.17686/) (2026)

## Abstract (arXiv)

The static ``train then deploy" paradigm fundamentally limits Large Language Models (LLMs) from dynamically adapting their weights in response to continuous streams of new information inherent in real-world tasks. Test-Time Training (TTT) offers a compelling alternative by updating a subset of model parameters (fast weights) at inference time, yet its potential in the current LLM ecosystem is hindered by critical barriers including architectural incompatibility, computational inefficiency and misaligned fast weight objectives for language modeling. In this work, we introduce In-Place Test-Time Training (In-Place TTT), a framework that seamlessly endows LLMs with Test-Time Training ability. In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling a ``drop-in" enhancement for LLMs without costly retraining from scratch. Furthermore, we replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective explicitly aligned with the Next-Token-Prediction task governing autoregressive language modeling. This principled objective, combined with an efficient chunk-wise update mechanism, results in a highly scalable algorithm compatible with context parallelism. Extensive experiments validate our framework's effectiveness: as an in-place enhancement, it enables a 4B-parameter model to achieve superior performance on tasks with contexts up to 128k, and when pretrained from scratch, it consistently outperforms competitive TTT-related approaches. Ablation study results further provide deeper insights on our design choices. Collectively, our results establish In-Place TTT as a promising step towards a paradigm of continual learning in LLMs.