mlsanityData to 14 Oct 2026

How Many Demonstrations Do You Need for In-context Learning?

arXiv 2303.08119 ↗First posted 14 Mar 2023cs.AIcited by 52

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are capable to perform complex reasoning by in-context learning (ICL) when provided with a few input-output demonstrations (demos) and more powerful when intermediate reasoning steps ("chain of thoughts (CoT)") of the demos are given. Is it necessary to use multi-demo in ICL? In this paper, we study ICL using fewer demos for each test query on the tasks in~citewei2022chain. Surprisingly, we do not observe significant degradation when using only one randomly chosen demo. To study this phenomenon, for each test query, we categorize demos into "correct demos" leading to the correct answer, and "wrong demos" resulting in wrong answers. Our analysis reveals an inherent bias in those widely studied datasets: most demos are correct for a majority of test queries, which explains the good performance of using one random demo. Moreover, ICL (with and w/o CoT) using only one correct demo significantly outperforms all-demo ICL adopted by most previous works, indicating the weakness of LLMs in finding correct demo(s) for input queries, which is difficult to evaluate on the biased datasets. Furthermore, we observe a counterintuitive behavior of ICL using multi-demo, i.e., its accuracy degrades(improves) when given more correct(wrong) demos. This implies that ICL can be easily misguided by interference among demos and their spurious correlations. Our analyses highlight several fundamental challenges that need to be addressed in LLMs training, ICL, and benchmark design.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 26 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

22 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  12. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
8 more
  1. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  5. 2016Solving General Arithmetic Word Problemscited by 360
  6. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  7. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  8. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 16; 2026: 6.

52 papers.

Sort:

By area:

  1. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  2. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  3. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  4. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  5. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  6. 2023Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Modelscited by 45
  7. 2023When do you need Chain-of-Thought Prompting for ChatGPT?cited by 35
  8. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
  9. 2024Automated Data Curation for Robust Language Model Fine-Tuningcited by 27
  10. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 19
  11. 2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 19
  12. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 15
38 more
  1. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  2. 2025Knowledge Graph-extended Retrieval Augmented Generation for Question Answeringcited by 14
  3. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  4. 2024The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysiscited by 13
  5. 2024In-Context Learning Demonstration Selection via Influence Analysiscited by 11
  6. 2024Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?cited by 11
  7. 2023GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Datasetcited by 7
  8. 2025When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languagescited by 7
  9. 2024Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanismcited by 6
  10. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  11. 2025From Evidence to Belief: A Bayesian Epistemology Approach to Language Modelscited by 6
  12. 2023Self-Specialization: Uncovering Latent Expertise within Large Language Modelscited by 5
  13. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  14. 2024Language Models Don't Learn the Physical Manifestation of Languagecited by 4
  15. 2024The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendationcited by 4
  16. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  17. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  18. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  19. 2024Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMscited by 3
  20. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  21. 2025Dynamic Retriever for In-Context Knowledge Editing via Policy Optimizationcited by 3
  22. 2023PACIT: Unlocking the Power of Examples for Better In-Context Instruction Tuningcited by 2
  23. 2024Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Taskscited by 2
  24. 2024Active Prompt Tuning Enables Gpt-40 To Do Efficient Classification Of Microscopy Imagescited by 2
  25. 2025LSC-Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM-Generated Synthetic Datacited by 2
  26. 2025Monte Carlo Sampling for Analyzing In-Context Examplescited by 2
  27. 2025ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoningcited by 2
  28. 2024ParaICL: Towards Parallel In-Context Learningcited by 1
  29. 2025Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsingcited by 1
  30. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  31. 2025On the Relationship Between the Choice of Representation and In-Context Learningcited by 1
  32. 2024Can AI Outperform Human Experts in Creating Social Media Creatives?cited by 0
  33. 2024SeCoKD: Aligning Large Language Models for In-Context Learning with Fewer Shotscited by 0
  34. 2025GRAD: Generative Retrieval-Aligned Demonstration Sampler for Efficient Few-Shot Reasoningcited by 0
  35. 2026Adaptive Data Augmentation with Multi-armed Bandit: Sample-Efficient Embedding Calibration for Implicit Pattern Recognitioncited by 0
  36. 2026Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognitioncited by 0
  37. 2026ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cuescited by 0
  38. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  2. 2023In-Context Learning with Iterative Demonstration Selection
  3. 2023Are Human-generated Demonstrations Necessary for In-context Learning?
  4. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  5. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
  6. 2024What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis
  7. 2026Demonstrations, CoT, and Prompting: A Theoretical Analysis of ICL
  8. 2025Refract ICL: Rethinking Example Selection in the Era of Million-Token Models