mlsanityData to 14 Oct 2026

The Role of Diversity in In-Context Learning for Large Language Models

arXiv 2505.19426 ↗First posted 26 May 2025cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a crucial capability of current large language models (LLMs), where the selection of examples plays a key role in performance. While most existing approaches focus on selecting the most similar examples to the query, the impact of diversity in example selection remains underexplored. We systematically investigate the role of diversity in in-context example selection through experiments across a range of tasks, from sentiment classification to more challenging math and code problems. Experiments on Llama-3.1, Gemma-2, and Mistral-v0.3 families of models show that diversity-aware selection methods improve performance, particularly on complex tasks like math and code, and enhance robustness to out-of-distribution queries. To support these findings, we introduce a theoretical framework that explains the benefits of incorporating diversity in in-context example selection.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 58 of its 73 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

58 of 73 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2023Mistral 7Bcited by 3,164
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
38 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  12. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  13. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  14. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  15. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  16. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  17. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  18. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  19. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  20. 2023The Learnability of In-Context Learningcited by 127
  21. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  22. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  23. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  24. 2023Finding Support Examples for In-Context Learningcited by 104
  25. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  26. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 84
  27. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  28. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  29. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 72
  30. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  31. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  32. 2022Data Curation Alone Can Stabilize In-context Learningcited by 59
  33. 2023Coverage-based Example Selection for In-Context Learningcited by 59
  34. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  35. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  36. 2023A Latent Space Theory for Emergent Abilities in Large Language Modelscited by 43
  37. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  38. 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 29

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Bridging Streaming Continual Learning via In-Context Large Tabular Modelscited by 7
  2. 2025Counterfactual reasoning: an analysis of in-context emergencecited by 4
  3. 2026Learning to Select Visual In-Context Demonstrationscited by 2
  4. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  5. 2026Design-Specification Tiling for ICL-based CAD Code Generationcited by 0
  6. 2026AdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program Repaircited by 0