mlsanityData to 7 Oct 2026

Coverage-based Example Selection for In-Context Learning

arXiv 2305.14907 ↗First posted 24 May 2023cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance. The standard approach of independently ranking and selecting the most similar examples selects redundant examples while omitting important information. In this work, we show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects, e.g. reasoning patterns, of the test input. We further extend BSR and many standard metrics to easily optimizable set-level metrics, giving still better coverage of those salient aspects. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, set selection using Set-BSR outperforms independent ranking by up to 17 points on average and, despite being training-free, surpasses methods that leverage task or LLM-specific training.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

39 of 67 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
27 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  5. 2023StarCoder: may the source be with you!cited by 790
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  8. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  9. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  10. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  11. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  12. 2022Exploring Length Generalization in Large Language Modelscited by 204
  13. 2022In-context Examples Selection for Machine Translationcited by 172
  14. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  15. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  16. 2023Compositional Exemplars for In-context Learningcited by 143
  17. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  18. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  19. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  20. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  21. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  22. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  23. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  24. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  25. 2022Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluationcited by 8
  26. 2021Question Decomposition with Dependency Graphscited by 7
  27. 2022Simplifying Semantic Annotations of SMCalFlowcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 19; 2025: 23; 2026: 11.

58 papers.

Sort:

By area:

  1. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  2. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  3. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  4. 2024Task Facet Learning: A Structured Approach to Prompt Optimizationcited by 17
  5. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  6. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  7. 2024Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model promptingcited by 12
  8. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  9. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learningcited by 10
  10. 2025Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Modelscited by 10
  11. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 9
  12. 2024Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Processcited by 9
38 more
  1. 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verificationcited by 8
  2. 2024Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparisoncited by 7
  3. 2024Speechworthy Instruction-tuned Language Modelscited by 7
  4. 2024Demonstration Selection for In-Context Learning via Reinforcement Learningcited by 7
  5. 2025Meta-Prompt Optimization for LLM-Based Sequential Decision Makingcited by 7
  6. 2023GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottleneckscited by 6
  7. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  8. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  9. 2025ProCut: LLM Prompt Compression via Attribution Estimationcited by 5
  10. 2025Leveraging In-Context Learning for Language Model Agentscited by 4
  11. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  12. 2025Topic Coverage-based Demonstration Retrieval for In-Context Learningcited by 3
  13. 2023Using Natural Language Explanations to Improve Robustness of In-context Learningcited by 2
  14. 2023Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learningcited by 2
  15. 2024Concept-aware Data Construction Improves In-context Learning of Language Modelscited by 2
  16. 2024Effective In-Context Example Selection through Data Compressioncited by 2
  17. 2025Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platformscited by 2
  18. 2025Incomplete In-context Learningcited by 2
  19. 2025Exploring Explanations Improves the Robustness of In-Context Learningcited by 2
  20. 2025Distilling Many-Shot In-Context Learning into a Cheat Sheetcited by 2
  21. 2026Optimal Skill Selection for LLM Agents with Provable Bicriteria Guaranteescited by 2
  22. 2024ParaICL: Towards Parallel In-Context Learningcited by 1
  23. 2024SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine Translationcited by 1
  24. 2025Learning to Substitute Components for Compositional Generalizationcited by 1
  25. 2025"Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selectioncited by 1
  26. 2025Diverse In-Context Example Selection After Decomposing Programs and Aligned Utterances Improves Semantic Parsingcited by 1
  27. 2025Boosting In-Context Learning in LLMs Through the Lens of Classical Supervised Learningcited by 1
  28. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  29. 2025Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQLcited by 1
  30. 2026Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrievalcited by 1
  31. 2026Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learningcited by 1
  32. 2026Towards Reliable, Generalizable, and Specific In-Context Knowledge Editing via Multi-Objective Reinforcement Learningcited by 1
  33. 2024Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correctioncited by 0
  34. 2024In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewritingcited by 0
  35. 2024What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicingcited by 0
  36. 2025Improving Dialogue State Tracking through Combinatorial Search for In-Context Examplescited by 0
  37. 2025Rethinking Reasoning in LLMs: Neuro-Symbolic Local RetoMaton Beyond ICL and CoTcited by 0
  38. 2025Retrieval and Augmentation of Domain Knowledge for Text-to-SQL Semantic Parsingcited by 0

50 most cited of 58 All 58 (JSON)