mlsanityData to 14 Oct 2026

Learning to (Learn at Test Time)

arXiv 2310.13807 ↗First posted 20 Oct 2023cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 38 of its 67 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

38 of 67 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2014Going Deeper with Convolutionscited by 4,241
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  11. 2022Transformers learn in-context by gradient descentcited by 732
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
26 more
  1. 2016Learning to learn by gradient descent by gradient descentcited by 620
  2. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  3. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  4. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  5. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 286
  8. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  9. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  10. 2020Consistent Video Depth Estimationcited by 220
  11. 2022Test-Time Training with Masked Autoencoderscited by 212
  12. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  13. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  14. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  15. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  16. 2013Learning by Transductioncited by 121
  17. 2017A Tutorial on Kernel Density Estimation and Recent Advancescited by 101
  18. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  19. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  20. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  21. 2023Transformers as Support Vector Machinescited by 74
  22. 2018Online Model Distillation for Efficient Video Inferencecited by 60
  23. 2020Meta Learning Backpropagation And Improving Itcited by 49
  24. 2018Meta-Learning Update Rules for Unsupervised Representation Learningcited by 37
  25. 2023Test-Time Training on Video Streamscited by 37
  26. 2022Meta-Learning Fast Weight Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 5; 2026: 9.

17 papers.

Sort:

By area:

  1. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  2. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  3. 2025End-to-End Test-Time Training for Long Contextcited by 62
  4. 2025SLOT: Sample-specific Language Model Optimization at Test-timecited by 18
  5. 2025Learning to Reason from Feedback at Test-Timecited by 10
  6. 2026AdaMEM: Test-Time Adaptive Memory for Language Agentscited by 6
  7. 2023MyPortrait: Morphable Prior-Guided Personalized Portrait Generationcited by 3
  8. 2025UrbanMind: Urban Dynamics Prediction with Multifaceted Spatial-Temporal Large Language Modelscited by 2
  9. 2026Learning to Remember, Learn, and Forget in Attention-Based Modelscited by 2
  10. 2026Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstructioncited by 1
  11. 2026The Power of Test-Time Training for Approximate Samplingcited by 1
  12. 2025Titans Revisited: A Lightweight Reimplementation and Critical Analysis of a Test-Time Memory Modelcited by 0
5 more
  1. 2026From Migration to Calibration: Aligning Agents with Deployment Requirementscited by 0
  2. 2026Universal Test-Time Trainingcited by 0
  3. 2026Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveriescited by 0
  4. 2026Towards In-Parameter Memory Augmentation for Large Language Modelscited by 0
  5. 2026Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agentscited by 0