# Coverage-based Example Selection for In-Context Learning

arXiv: https://arxiv.org/abs/2305.14907 · HTML: https://mlsanity.com/paper/2305.14907/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Exploring Length Generalization in Large Language Models](https://mlsanity.com/paper/2207.04901/) (2022)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Compositional Exemplars for In-context Learning](https://mlsanity.com/paper/2302.05698/) (2023)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)

## Built on by (most cited first, 30 of 59)

- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars](https://mlsanity.com/paper/2405.16122/) (2024)
- [Task Facet Learning: A Structured Approach to Prompt Optimization](https://mlsanity.com/paper/2406.10504/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Understanding and Improving Information Preservation in Prompt Compression for LLMs](https://mlsanity.com/paper/2503.19114/) (2025)
- [Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting](https://mlsanity.com/paper/2402.12801/) (2024)
- [From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning](https://mlsanity.com/paper/2407.00902/) (2024)
- [Leveraging Code to Improve In-context Learning for Semantic Parsing](https://mlsanity.com/paper/2311.09519/) (2023)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)
- [Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models](https://mlsanity.com/paper/2503.07329/) (2025)
- [Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process](https://mlsanity.com/paper/2408.02103/) (2024)
- [From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification](https://mlsanity.com/paper/2403.06326/) (2024)
- [Demonstration Selection for In-Context Learning via Reinforcement Learning](https://mlsanity.com/paper/2412.03966/) (2024)
- [Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison](https://mlsanity.com/paper/2404.02835/) (2024)
- [Speechworthy Instruction-tuned Language Models](https://mlsanity.com/paper/2409.14672/) (2024)
- [Meta-Prompt Optimization for LLM-Based Sequential Decision Making](https://mlsanity.com/paper/2502.00728/) (2025)
- [GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks](https://mlsanity.com/paper/2311.09606/) (2023)
- [OptiSeq: Ordering Examples On-The-Fly for In-Context Learning](https://mlsanity.com/paper/2501.15030/) (2025)
- [The Role of Diversity in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2505.19426/) (2025)
- [ProCut: LLM Prompt Compression via Attribution Estimation](https://mlsanity.com/paper/2508.02053/) (2025)
- [Leveraging In-Context Learning for Language Model Agents](https://mlsanity.com/paper/2506.13109/) (2025)
- [Unraveling Indirect In-Context Learning Using Influence Functions](https://mlsanity.com/paper/2501.01473/) (2025)
- [Topic Coverage-based Demonstration Retrieval for In-Context Learning](https://mlsanity.com/paper/2509.12451/) (2025)
- [Using Natural Language Explanations to Improve Robustness of In-context Learning](https://mlsanity.com/paper/2311.07556/) (2023)
- [Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning](https://mlsanity.com/paper/2311.09619/) (2023)
- [Concept-aware Data Construction Improves In-context Learning of Language Models](https://mlsanity.com/paper/2403.09703/) (2024)
- [Effective In-Context Example Selection through Data Compression](https://mlsanity.com/paper/2405.11465/) (2024)
- [Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platforms](https://mlsanity.com/paper/2501.13977/) (2025)
- [Incomplete In-context Learning](https://mlsanity.com/paper/2505.07251/) (2025)

## Abstract (arXiv)

In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance. The standard approach of independently ranking and selecting the most similar examples selects redundant examples while omitting important information. In this work, we show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects, e.g. reasoning patterns, of the test input. We further extend BSR and many standard metrics to easily optimizable set-level metrics, giving still better coverage of those salient aspects. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, set selection using Set-BSR outperforms independent ranking by up to 17 points on average and, despite being training-free, surpasses methods that leverage task or LLM-specific training.