mlsanityData to 7 Oct 2026

In-context Learning and Gradient Descent Revisited

arXiv 2311.07772 ↗First posted 13 Nov 2023cs.CL cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. A recent line of work suggests that ICL performs gradient descent (GD)-based optimization implicitly. While appealing, much of the research focuses on simplified settings, where the parameters of a shallow model are optimized. In this work, we revisit evidence for ICL-GD correspondence on realistic NLP tasks and models. We find gaps in evaluation, both in terms of problematic metrics and insufficient baselines. We show that surprisingly, even untrained models achieve comparable ICL-GD similarity scores despite not exhibiting ICL. Next, we explore a major discrepancy in the flow of information throughout the model between ICL and GD, which we term Layer Causality. We propose a simple GD-based optimization procedure that respects layer causality, and show it improves similarity scores significantly.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

15 of 28 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  9. 2023In-Context Learning Creates Task Vectorscited by 302
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
3 more
  1. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  2. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  3. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  2. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  3. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  4. 2024In-Context Learning State Vector with Inner and Momentum Optimizationcited by 15
  5. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 8
  6. 2024Memorization in In-Context Learningcited by 6
  7. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  8. 2025LiteraryQA: Towards Effective Evaluation of Long-document Narrative QAcited by 6
  9. 2024Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Modelscited by 4
  10. 2025Counterfactual reasoning: an analysis of in-context emergencecited by 4
  11. 2025CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrationscited by 2
  12. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learningcited by 2
8 more
  1. 2025Context Tuning for In-Context Optimizationcited by 2
  2. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learningcited by 1
  3. 2025Supervised learning pays attentioncited by 1
  4. 2025What Neuroscience Can Teach AI About Learning in Continuously Changing Environmentscited by 0
  5. 2026The Alchemy of Thought: Understanding In-Context Learning Through Supervised Classificationcited by 0
  6. 2026Functional Subspace, where language models can use vector algebra to solve problemscited by 0
  7. 2026Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classificationcited by 0
  8. 2026Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transportcited by 0