# Learning to (Learn at Test Time) arXiv: https://arxiv.org/abs/2310.13807 ยท HTML: https://mlsanity.com/paper/2310.13807/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 67 references that are in the map. - First posted on arXiv 20 Oct 2023. ## Builds on (most cited first, 30 of 38) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021) - [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020) - [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020) - [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016) - [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022) - [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022) - [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016) - [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017) - [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021) - [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023) - [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023) - [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023) - ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017) - [Transformers learn to implement preconditioned gradient descent for in-context learning](https://mlsanity.com/paper/2306.00297/) (2023) - [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022) - [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020) - [Test-Time Training with Masked Autoencoders](https://mlsanity.com/paper/2209.07522/) (2022) - [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016) - [One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention](https://mlsanity.com/paper/2307.03576/) (2023) - [SOFT: Softmax-free Transformer with Linear Complexity](https://mlsanity.com/paper/2110.11945/) (2021) - [Self-Supervised Policy Adaptation during Deployment](https://mlsanity.com/paper/2007.04309/) (2020) - [Learning by Transduction](https://mlsanity.com/paper/1301.7375/) (2013) - [A Tutorial on Kernel Density Estimation and Recent Advances](https://mlsanity.com/paper/1704.03924/) (2017) - [Better plain ViT baselines for ImageNet-1k](https://mlsanity.com/paper/2205.01580/) (2022) ## Built on by (most cited first, 17 of 17) - [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024) - [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024) - [End-to-End Test-Time Training for Long Context](https://mlsanity.com/paper/2512.23675/) (2025) - [SLOT: Sample-specific Language Model Optimization at Test-time](https://mlsanity.com/paper/2505.12392/) (2025) - [Learning to Reason from Feedback at Test-Time](https://mlsanity.com/paper/2502.15771/) (2025) - [AdaMEM: Test-Time Adaptive Memory for Language Agents](https://mlsanity.com/paper/2606.05684/) (2026) - [MyPortrait: Morphable Prior-Guided Personalized Portrait Generation](https://mlsanity.com/paper/2312.02703/) (2023) - [UrbanMind: Urban Dynamics Prediction with Multifaceted Spatial-Temporal Large Language Models](https://mlsanity.com/paper/2505.11654/) (2025) - [Learning to Remember, Learn, and Forget in Attention-Based Models](https://mlsanity.com/paper/2602.09075/) (2026) - [Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction](https://mlsanity.com/paper/2605.16981/) (2026) - [The Power of Test-Time Training for Approximate Sampling](https://mlsanity.com/paper/2606.11437/) (2026) - [Titans Revisited: A Lightweight Reimplementation and Critical Analysis of a Test-Time Memory Model](https://mlsanity.com/paper/2510.09551/) (2025) - [From Migration to Calibration: Aligning Agents with Deployment Requirements](https://mlsanity.com/paper/2609.35149/) (2026) - [Universal Test-Time Training](https://mlsanity.com/paper/2610.05484/) (2026) - [Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries](https://mlsanity.com/paper/2610.06269/) (2026) - [Towards In-Parameter Memory Augmentation for Large Language Models](https://mlsanity.com/paper/2610.08630/) (2026) - [Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents](https://mlsanity.com/paper/2610.13058/) (2026) ## Abstract (arXiv) We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run.