Data to 7 Oct 2026

Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers

arXiv 2212.10559 ↗First posted 20 Dec 2022cs.CLcited by 204

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without parameter updates. Despite the great success in performance, its working mechanism still remains an open question. In this paper, we explain language models as meta-optimizers and understand in-context learning as implicit finetuning. Theoretically, we figure out that Transformer attention has a dual form of gradient descent. On top of it, we understand ICL as follows: GPT first produces meta-gradients according to the demonstration examples, and then these meta-gradients are applied to the original GPT to build an ICL model. We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding. Experimental results show that in-context learning behaves similarly to explicit finetuning from multiple perspectives. Inspired by the dual form between Transformer attention and gradient descent, we design a momentum-based attention by analogy with gradient descent with momentum. The improved performance over vanilla attention further supports our understanding from another perspective, and more importantly, shows the potential to utilize our understanding for future model design. The code is available at urlhttps://aka.ms/icl.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

14 of 29 references.

References · 14

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
2015Character-level Convolutional Networks for Text Classificationcited by 2,293
2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
2022In-context Learning and Induction Headscited by 930
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
2022Transformers learn in-context by gradient descentcited by 714
2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102

2 more

2020Meta Learning Backpropagation And Improving Itcited by 49
2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 60; 2025: 71; 2026: 30.

204 papers.

Sort:

By area:

2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
2023Challenges and Applications of Large Language Modelscited by 251
2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
2023In-Context Convergence of Transformerscited by 112
2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
2023In-Context Learning with Iterative Demonstration Selectioncited by 74

38 more

2023Efficient Benchmarking of Language Modelscited by 71
2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 63
2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
2025Learning without training: The implicit dynamics of in-context learningcited by 41
2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 36
2025Emergent Abilities in Large Language Models: A Surveycited by 34
2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activationscited by 32
2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
2023Understanding In-Context Learning from Repetitionscited by 27
2024Cycles of Thought: Measuring LLM Confidence through Stable Explanationscited by 27
2024GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Modelcited by 27
2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
2023LLMs as Hackers: Autonomous Linux Privilege Escalation Attackscited by 25
2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignmentcited by 25
2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 23
2024In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitznesscited by 22
2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
2024In-context Time Series Predictorcited by 21
2023In-context Learning and Gradient Descent Revisitedcited by 20
2024Do Efficient Transformers Really Save Computation?cited by 20
2023Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learningcited by 19
2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
2023Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioningcited by 18
2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
2024Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problemcited by 17

50 most cited of 204 All 204 (JSON)