# A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt

arXiv: https://arxiv.org/abs/2606.15569 · HTML: https://mlsanity.com/paper/2606.15569/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 79 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Parameter-free Online Test-time Adaptation](https://mlsanity.com/paper/2201.05718/) (2022)
- [Robust Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2303.13899/) (2023)
- [Test-Time Training Done Right](https://mlsanity.com/paper/2505.23884/) (2025)
- [PAC-Bayesian Theory Meets Bayesian Inference](https://mlsanity.com/paper/1605.08636/) (2016)
- [Test-Time Training on Nearest Neighbors for Large Language Models](https://mlsanity.com/paper/2305.18466/) (2023)
- [Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypes](https://mlsanity.com/paper/2207.11707/) (2022)
- [Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs](https://mlsanity.com/paper/2410.08020/) (2024)
- [What can a Single Attention Layer Learn? A Study Through the Random Features Lens](https://mlsanity.com/paper/2307.11353/) (2023)
- [On-the-Fly Test-time Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/2203.05574/) (2022)
- [Can Transformers Learn Full Bayesian Inference in Context?](https://mlsanity.com/paper/2501.16825/) (2025)
- [Amortized Variational Inference: When and Why?](https://mlsanity.com/paper/2307.11018/) (2023)
- [In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning](https://mlsanity.com/paper/2510.10981/) (2025)
- [Goal-Oriented Optimal Design of Experiments for Large-Scale Bayesian Linear Inverse Problems](https://mlsanity.com/paper/1802.06517/) (2018)
- [Test-Time Training Provably Improves Transformers as In-context Learners](https://mlsanity.com/paper/2503.11842/) (2025)
- [Amortized In-Context Bayesian Posterior Estimation](https://mlsanity.com/paper/2502.06601/) (2025)
- [Continual Test-time Adaptation for End-to-end Speech Recognition on Noisy Speech](https://mlsanity.com/paper/2406.11064/) (2024)
- [A Layer Selection Approach to Test Time Adaptation](https://mlsanity.com/paper/2404.03784/) (2024)
- [Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models](https://mlsanity.com/paper/2509.24510/) (2025)
- [Test time training enhances in-context learning of nonlinear functions](https://mlsanity.com/paper/2509.25741/) (2025)

## Abstract (arXiv)

Test-time training (TTT) adapts a pretrained model to each prompt via parameter updates, improving accuracy under pretraining-to-test distribution shifts. Yet, its performance often suffers from instability and sensitivity to hyperparameters such as update steps and subspace. We explain this behavior through a decision-theoretic lens, treating TTT as implicit Bayesian inference in the kernel regime. Under a Gaussian process benchmark, we show that TTT reduces prediction error when updates are spectrally matched to the prompt's signal-to-noise ratio and aligned with query-relevant eigen-directions. This perspective underpins the following results: (1) we show when fixed update steps and subspaces fail under distribution shifts, motivating adaptive strategies; (2) we prove that selecting update steps via prompt evidence admits a PAC-Bayes guarantee against overfitting; and (3) we characterize the Bayes-optimal update subspace under a linear-Gaussian correction model, yielding a scoring rule for selecting Transformer blocks and heads. Our theory helps explain the empirical instability of TTT, taking a step toward principled guidance for when, how far, and which directions to adapt.