# Revisiting Demonstration Selection Strategies in In-Context Learning

arXiv: https://arxiv.org/abs/2401.12087 · HTML: https://mlsanity.com/paper/2401.12087/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering](https://mlsanity.com/paper/2212.10375/) (2022)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models](https://mlsanity.com/paper/2303.13809/) (2023)
- [Intention Analysis Makes LLMs A Good Jailbreak Defender](https://mlsanity.com/paper/2401.06561/) (2024)
- [A Retrieve-and-Edit Framework for Predicting Structured Outputs](https://mlsanity.com/paper/1812.01194/) (2018)
- [Dr.ICL: Demonstration-Retrieved In-context Learning](https://mlsanity.com/paper/2305.14128/) (2023)
- [OpenICL: An Open-Source Framework for In-context Learning](https://mlsanity.com/paper/2303.02913/) (2023)
- [In-Context Demonstration Selection with Cross Entropy Difference](https://mlsanity.com/paper/2305.14726/) (2023)
- [Vega-MT: The JD Explore Academy Translation System for WMT22](https://mlsanity.com/paper/2209.09444/) (2022)
- [Improving Neural Machine Translation by Denoising Training](https://mlsanity.com/paper/2201.07365/) (2022)

## Built on by (most cited first, 30 of 58)

- [TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systems](https://mlsanity.com/paper/2408.09199/) (2024)
- [Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation](https://mlsanity.com/paper/2402.13408/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning](https://mlsanity.com/paper/2502.15401/) (2025)
- [Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation](https://mlsanity.com/paper/2508.19999/) (2025)
- [PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs](https://mlsanity.com/paper/2410.06704/) (2024)
- [What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis](https://mlsanity.com/paper/2412.12157/) (2024)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [Does Few-Shot Learning Help LLM Performance in Code Synthesis?](https://mlsanity.com/paper/2412.02906/) (2024)
- [Investigating the Zone of Proximal Development of Language Models for In-Context Learning](https://mlsanity.com/paper/2502.06990/) (2025)
- [Retrieval-augmented in-context learning for multimodal large language models in disease classification](https://mlsanity.com/paper/2505.02087/) (2025)
- [Adaptive Task Vectors for Large Language Models](https://mlsanity.com/paper/2506.03426/) (2025)
- [Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning](https://mlsanity.com/paper/2602.02099/) (2026)
- [Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding](https://mlsanity.com/paper/2502.13738/) (2025)
- [TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistency](https://mlsanity.com/paper/2502.19163/) (2025)
- [Training-free LLM Verification via Recycling Few-shot Examples](https://mlsanity.com/paper/2506.17251/) (2025)
- [From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When](https://mlsanity.com/paper/2406.00131/) (2024)
- [Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting](https://mlsanity.com/paper/2410.01724/) (2024)
- [Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models](https://mlsanity.com/paper/2501.07892/) (2025)
- [Topic Coverage-based Demonstration Retrieval for In-Context Learning](https://mlsanity.com/paper/2509.12451/) (2025)
- [RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplars](https://mlsanity.com/paper/2502.11681/) (2025)
- [Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection](https://mlsanity.com/paper/2506.03476/) (2025)
- [Surprise Calibration for Better In-Context Learning](https://mlsanity.com/paper/2506.12796/) (2025)
- [InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning](https://mlsanity.com/paper/2509.12263/) (2025)
- [Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data](https://mlsanity.com/paper/2407.16516/) (2024)
- [D.Va: Validate Your Demonstration First Before You Use It](https://mlsanity.com/paper/2502.13646/) (2025)
- [Multi-Layer Attention is the Amplifier of Demonstration Effectiveness](https://mlsanity.com/paper/2508.00385/) (2025)
- [From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs](https://mlsanity.com/paper/2509.23196/) (2025)
- [Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspective](https://mlsanity.com/paper/2510.21999/) (2025)
- [Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing](https://mlsanity.com/paper/2511.12661/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and it is still unclear why this happens or what factors will influence its choice. In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We further proposed a data- and model-dependent demonstration selection method, TopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples, resulting in a simple and effective recipe for ICL. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code will be released.