# In-context Learning and Gradient Descent Revisited

arXiv: https://arxiv.org/abs/2311.07772 · HTML: https://mlsanity.com/paper/2311.07772/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Transformers learn to implement preconditioned gradient descent for in-context learning](https://mlsanity.com/paper/2306.00297/) (2023)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention](https://mlsanity.com/paper/2202.05798/) (2022)

## Built on by (most cited first, 20 of 20)

- [From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples](https://mlsanity.com/paper/2404.07544/) (2024)
- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning](https://mlsanity.com/paper/2505.12212/) (2025)
- [In-Context Learning State Vector with Inner and Momentum Optimization](https://mlsanity.com/paper/2404.11225/) (2024)
- [Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations](https://mlsanity.com/paper/2506.09048/) (2025)
- [Memorization in In-Context Learning](https://mlsanity.com/paper/2408.11546/) (2024)
- [The Role of Diversity in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2505.19426/) (2025)
- [LiteraryQA: Towards Effective Evaluation of Long-document Narrative QA](https://mlsanity.com/paper/2510.13494/) (2025)
- [Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models](https://mlsanity.com/paper/2409.04318/) (2024)
- [CoT-ICL Lab: A Synthetic Framework for Studying Chain-of-Thought Learning from In-Context Demonstrations](https://mlsanity.com/paper/2502.15132/) (2025)
- [Counterfactual reasoning: an analysis of in-context emergence](https://mlsanity.com/paper/2506.05188/) (2025)
- [Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning](https://mlsanity.com/paper/2505.11004/) (2025)
- [Context Tuning for In-Context Optimization](https://mlsanity.com/paper/2507.04221/) (2025)
- [Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning](https://mlsanity.com/paper/2506.11516/) (2025)
- [Supervised learning pays attention](https://mlsanity.com/paper/2512.09912/) (2025)
- [What Neuroscience Can Teach AI About Learning in Continuously Changing Environments](https://mlsanity.com/paper/2507.02103/) (2025)
- [The Alchemy of Thought: Understanding In-Context Learning Through Supervised Classification](https://mlsanity.com/paper/2601.01290/) (2026)
- [Functional Subspace, where language models can use vector algebra to solve problems](https://mlsanity.com/paper/2602.01687/) (2026)
- [Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification](https://mlsanity.com/paper/2604.11613/) (2026)
- [Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport](https://mlsanity.com/paper/2608.11342/) (2026)

## Abstract (arXiv)

In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. A recent line of work suggests that ICL performs gradient descent (GD)-based optimization implicitly. While appealing, much of the research focuses on simplified settings, where the parameters of a shallow model are optimized. In this work, we revisit evidence for ICL-GD correspondence on realistic NLP tasks and models. We find gaps in evaluation, both in terms of problematic metrics and insufficient baselines. We show that surprisingly, even untrained models achieve comparable ICL-GD similarity scores despite not exhibiting ICL. Next, we explore a major discrepancy in the flow of information throughout the model between ICL and GD, which we term Layer Causality. We propose a simple GD-based optimization procedure that respects layer causality, and show it improves similarity scores significantly.