mlsanityData to 14 Oct 2026

Learning without training: The implicit dynamics of in-context learning

arXiv 2507.16003 ↗First posted 21 Jul 2025cs.CL cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theoretical analysis and experimentation that this simple mechanism may help explain why LLMs demonstrate capabilities of in-context learning, beyond what is captured during training. Specifically, we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal low-rank update representing the context.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 21 Jul 2025.

Builds on

44 of 57 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
32 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2022Transformers learn in-context by gradient descentcited by 732
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  4. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  7. 2022Memory-Based Model Editing at Scalecited by 447
  8. 2023Larger language models do in-context learning differentlycited by 394
  9. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  10. 2023In-Context Learning Creates Task Vectorscited by 312
  11. 2023Function Vectors in Large Language Modelscited by 308
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  13. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  14. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  15. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  16. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  17. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  18. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  19. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  20. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  21. 2020Meta Learning Backpropagation And Improving Itcited by 49
  22. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 39
  23. 2023Simplifying Transformer Blockscited by 37
  24. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35
  25. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  26. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
  27. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  28. 2025Task Vectors in In-Context Learning: Emergence, Formation, and Benefitcited by 20
  29. 2024Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformerscited by 7
  30. 2024In-Context Deep Learning via Transformer Modelscited by 6
  31. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  32. 2025Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspectivecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 29.

43 papers.

Sort:

By area:

  1. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 129
  2. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 18
  3. 2026GENIUS: Generative Fluid Intelligence Evaluation Suitecited by 12
  4. 2025Think-on-Graph 3.0: Efficient and Adaptive LLM Reasoning on Heterogeneous Graphs via Multi-Agent Dual-Evolving Context Retrievalcited by 10
  5. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  6. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamicscited by 8
  7. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 7
  8. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  9. 2025Equivalence of Context and Parameter Updates in Modern Transformer Blockscited by 5
  10. 2026Steer Like the LLM: Activation Steering that Mimics Promptingcited by 5
  11. 2026Linguistics and Human Brain: A Perspective of Computational Neurosciencecited by 4
  12. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
31 more
  1. 2025Transmuting prompts into weightscited by 3
  2. 2025The Narrative Continuity Test: A Conceptual Framework for Evaluating Identity Persistence in AI Systemscited by 3
  3. 2026Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Modelscited by 3
  4. 2026Weight Updates as Activation Shifts: A Principled Framework for Steeringcited by 3
  5. 2026Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agentscited by 3
  6. 2026Fast Spatial Memory with Elastic Test-Time Trainingcited by 3
  7. 2025Instruction-Level Weight Shaping: A Framework for Self-Improving AI Agentscited by 2
  8. 2025Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptationcited by 2
  9. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  10. 2026Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selectioncited by 2
  11. 2026Large language models reorganize representational geometry during in-context learningcited by 2
  12. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 2
  13. 2026Locating and Controlling Implicit Personalization in Large Language Modelscited by 2
  14. 2025A Criminology of Machinescited by 1
  15. 2025HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoningcited by 1
  16. 2025A Simple Generalisation of the Implicit Dynamics of In-Context Learningcited by 1
  17. 2026Training Large Reasoning Models Efficiently via Progressive Thought Encodingcited by 1
  18. 2026Evaluating Temporal Consistency in Multi-Turn Language Modelscited by 1
  19. 2026Hypothesis generation and updating in large language modelscited by 1
  20. 2026Prompting Complexity: Shortest Prompts for Texts and Behaviors in LLMscited by 1
  21. 2026PatchKV: Weight-Space Compensation of KV Cachecited by 1
  22. 2025Few-Shot and Training-Free Review Generation via Conversational Promptingcited by 0
  23. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
  24. 2026Mitigating Many-shot Jailbreak Attacks with One Single Demonstrationcited by 0
  25. 2026Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learningcited by 0
  26. 2026Graph-based Target Back-Propagation for Context Adaptation in Multi-LLM Agentic Systemscited by 0
  27. 2026Syntax vs. Semantics: How Transformers Learn Deep Dependenciescited by 0
  28. 2026From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoringcited by 0
  29. 2026The Safety Operator: Modulating the Expression of Safety Instructions via Spectral Optimizationcited by 0
  30. 2026When Can Prefixes Compile LoRA? Exact Resource-Capped Tests for Frozen Attentioncited by 0
  31. 2026COMPASS: Finding Where Reasoning Lives in Language Modelscited by 0