mlsanityData to 7 Oct 2026

An Explanation of In-context Learning as Implicit Bayesian Inference

arXiv 2111.02080 ↗First posted 3 Nov 2021cs.CL cs.LGcited by 915

Not on the map. Why?

Cited by 87 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples. The LM learns from these examples without being explicitly pretrained to learn. Thus, it is unclear what enables in-context learning. In this paper, we study how in-context learning can emerge when pretraining documents have long-range coherence. Here, the LM must infer a latent document-level concept to generate coherent next tokens during pretraining. At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt. We prove when this occurs despite a distribution mismatch between prompts and pretraining data in a setting where the pretraining distribution is a mixture of HMMs. In contrast to messy large-scale datasets used to train LMs capable of in-context learning, we generate a small-scale synthetic dataset (GINC) where Transformers and LSTMs both exhibit in-context learning. Beyond the theory, experiments on GINC exhibit large-scale real-world phenomena including improved in-context performance with model scaling (despite the same pretraining loss), sensitivity to example order, and instances where zero-shot is better than few-shot in-context learning.

Key facts. Cited by 915 papers in the map (data through 7 Oct 2026). Builds on 23 of its 44 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

23 of 44 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
11 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  8. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  9. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  10. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  11. 2016Multiclass Classification Calibration Functionscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 48; 2023: 216; 2024: 267; 2025: 193; 2026: 190.

915 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023Large Language Modelscited by 774
  9. 2022A Survey on In-context Learningcited by 717
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
38 more
  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  3. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  4. 2023Larger language models do in-context learning differentlycited by 386
  5. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  6. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  7. 2023In-Context Learning Creates Task Vectorscited by 302
  8. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  9. 2023Function Vectors in Large Language Modelscited by 292
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  13. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  14. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  15. 2023Challenges and Applications of Large Language Modelscited by 251
  16. 2022Can language models learn from explanations in context?cited by 248
  17. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  18. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  19. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  20. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  21. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  22. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  23. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  24. 2024Many-Shot In-Context Learningcited by 203
  25. 2023Large Language Models as General Pattern Machinescited by 196
  26. 2022Active Example Selection for In-Context Learningcited by 195
  27. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  28. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  29. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  30. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  31. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  32. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  33. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  34. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  35. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  36. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  37. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  38. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137

50 most cited of 915 All 915 (JSON)