mlsanityData to 14 Oct 2026

Test-Time Training on Nearest Neighbors for Large Language Models

arXiv 2305.18466 ↗First posted 29 May 2023cs.CL cs.LGcited by 88

Not on the map. Why?

Abstract (arXiv)

Many recent efforts augment language models with retrieval, by adding retrieved data to the input context. For this approach to succeed, the retrieved data must be added at both training and test time. Moreover, as input length grows linearly with the size of retrieved data, cost in computation and memory grows quadratically for modern Transformers. To avoid these complications, we simply fine-tune the model on retrieved data at test time, using its standard training setup. We build a large-scale distributed index based on text embeddings of the Pile dataset. For each test input, our system retrieves its neighbors and fine-tunes the model on their text. Surprisingly, retrieving and training on as few as 20 neighbors, each for only one gradient iteration, drastically improves performance across more than 20 language modeling tasks in the Pile. For example, test-time training with nearest neighbors significantly narrows the performance gap between a small GPT-2 and a GPT-Neo model more than 10 times larger. Sufficient index quality and size, however, are necessary. Our work establishes a first baseline of test-time training for language modeling.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

19 of 34 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  12. 2022Test-Time Training with Masked Autoencoderscited by 212
7 more
  1. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  2. 2013Learning by Transductioncited by 121
  3. 2021Efficient Nearest Neighbor Language Modelscited by 93
  4. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  5. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  6. 2017Dynamic Evaluation of Neural Sequence Modelscited by 55
  7. 2019Dynamic Evaluation of Transformer Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 39; 2026: 36.

88 papers.

Sort:

By area:

  1. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  2. 2026Reinforcement Learning via Self-Distillationcited by 332
  3. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  4. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  5. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  6. 2025End-to-End Test-Time Training for Long Contextcited by 62
  7. 2025Test-Time Learning for Large Language Modelscited by 50
  8. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 43
  9. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  10. 2023Test-Time Training on Video Streamscited by 37
  11. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
  12. 2025Self-Improving LLM Agents at Test-Timecited by 30
38 more
  1. 2025Cognitive Memory in Large Language Modelscited by 29
  2. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 28
  3. 2025Test3R: Learning to Reconstruct 3D at Test Timecited by 20
  4. 2025Let's (not) just put things in Context: Test-Time Training for Long-Context LLMscited by 18
  5. 2023Learning to (Learn at Test Time)cited by 17
  6. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  7. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  8. 2025Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiencescited by 15
  9. 2025Continuous Self-Improvement of Large Language Models by Test-time Training with Verifier-Driven Sample Selectioncited by 13
  10. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 13
  11. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  12. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  13. 2024Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approachcited by 11
  14. 2023Model-free Test Time Adaptation for Out-Of-Distribution Detectioncited by 9
  15. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  16. 2025Local Mixtures of Experts: Essentially Free Test-Time Training via Model Mergingcited by 9
  17. 2025Learning with Calibration: Exploring Test-Time Computing of Spatio-Temporal Forecastingcited by 9
  18. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  19. 2026Tool Verification for Test-Time Reinforcement Learningcited by 9
  20. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  21. 2026Workspace Optimization: How to Train Your Agentcited by 7
  22. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  23. 2025Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Modelscited by 6
  24. 2024No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representationscited by 5
  25. 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Modelscited by 5
  26. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 4
  27. 2025Test-time Correlation Alignmentcited by 4
  28. 2025Reusing Pre-Training Data at Test Time is a Compute Multipliercited by 4
  29. 2026Panini: Continual Learning in Token Space via Structured Memorycited by 4
  30. 2026No Time Like the Present: Agentic Test-Time Training for LLM Agentscited by 4
  31. 2026AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?cited by 4
  32. 2025Understanding and Mitigating Distribution Shifts For Machine Learning Force Fieldscited by 3
  33. 2025Few-Shot Segmentation of Historical Maps via Linear Probing of Vision Foundation Modelscited by 3
  34. 2025Context Tuning for In-Context Optimizationcited by 3
  35. 2025Rewiring Experts on the Fly:Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert modelscited by 3
  36. 2025Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibriumcited by 3
  37. 2026Learning Cross-View Object Correspondence via Cycle-Consistent Mask Predictioncited by 3
  38. 2024Active Few-Shot Fine-Tuningcited by 2

50 most cited of 88 All 88 (JSON)