mlsanityData to 7 Oct 2026

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

arXiv 2202.12837 ↗First posted 25 Feb 2022cs.CL cs.AIcited by 1,413

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.

Key facts. Cited by 1,413 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 46 of its 73 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

46 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoning15 Feb

Other references · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
33 more
  1. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  13. 2021MetaICL: Learning to Learn In Contextcited by 450
  14. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  15. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  16. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  17. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  18. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  19. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  20. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  21. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  22. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  23. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  24. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  25. 2021Meta-learning via Language Model In-context Tuningcited by 163
  26. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  27. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  28. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  29. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  30. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  31. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  32. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  33. 2018QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationshipscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 96; 2023: 385; 2024: 440; 2025: 270; 2026: 222.

1,413 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2022In-context Learning and Induction Headscited by 930
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  12. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
38 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2022A Survey on In-context Learningcited by 717
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  13. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  14. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  15. 2023Augmented Language Models: a Surveycited by 398
  16. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  17. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  18. 2023Larger language models do in-context learning differentlycited by 386
  19. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  20. 2023GPTScore: Evaluate as You Desirecited by 357
  21. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  22. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  23. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  24. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  25. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  26. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  27. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  28. 2023Function Vectors in Large Language Modelscited by 292
  29. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  30. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  31. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  32. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  33. 2022Prompting GPT-3 To Be Reliablecited by 258
  34. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  35. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  36. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  37. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  38. 2022Can language models learn from explanations in context?cited by 248

50 most cited of 1,413 All 1,413 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging In-Context Learning for Language Model Agents
  2. 2023Are Human-generated Demonstrations Necessary for In-context Learning?
  3. 2026CL-bench: A Benchmark for Context Learning
  4. 2024What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis
  5. 2024Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions
  6. 2024Focused Large Language Models are Stable Many-Shot Learners
  7. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learning
  8. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines