mlsanityData to 14 Oct 2026

Revisiting Demonstration Selection Strategies in In-Context Learning

arXiv 2401.12087 ↗First posted 22 Jan 2024cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and it is still unclear why this happens or what factors will influence its choice. In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We further proposed a data- and model-dependent demonstration selection method, TopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples, resulting in a simple and effective recipe for ICL. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code will be released.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

28 of 49 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
16 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  2. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  3. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  4. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  5. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  6. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  7. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  8. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  9. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  10. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  11. 2018A Retrieve-and-Edit Framework for Predicting Structured Outputscited by 73
  12. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 72
  13. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  14. 2023In-Context Demonstration Selection with Cross Entropy Differencecited by 17
  15. 2022Vega-MT: The JD Explore Academy Translation System for WMT22cited by 9
  16. 2022Improving Neural Machine Translation by Denoising Trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 28; 2026: 20.

58 papers.

Sort:

By area:

  1. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  2. 2024Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultationcited by 17
  3. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 16
  4. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  5. 2025Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimationcited by 8
  6. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  7. 2024What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysiscited by 7
  8. 2025What Makes a Good Natural Language Prompt?cited by 7
  9. 2024Does Few-Shot Learning Help LLM Performance in Code Synthesis?cited by 6
  10. 2025Investigating the Zone of Proximal Development of Language Models for In-Context Learningcited by 6
  11. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 5
  12. 2025Adaptive Task Vectors for Large Language Modelscited by 5
38 more
  1. 2026Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoningcited by 5
  2. 2025Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decodingcited by 4
  3. 2025TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistencycited by 4
  4. 2025Training-free LLM Verification via Recycling Few-shot Examplescited by 4
  5. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  6. 2024Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Promptingcited by 3
  7. 2025Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Modelscited by 3
  8. 2025Topic Coverage-based Demonstration Retrieval for In-Context Learningcited by 3
  9. 2025RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplarscited by 2
  10. 2025Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detectioncited by 2
  11. 2025Surprise Calibration for Better In-Context Learningcited by 2
  12. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  13. 2024Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Datacited by 1
  14. 2025D.Va: Validate Your Demonstration First Before You Use Itcited by 1
  15. 2025Multi-Layer Attention is the Amplifier of Demonstration Effectivenesscited by 1
  16. 2025From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMscited by 1
  17. 2025Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspectivecited by 1
  18. 2025Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editingcited by 1
  19. 2026Predicting Startup Success Using Large Language Models: A Novel In-Context Learning Approachcited by 1
  20. 2026Regime-aware financial volatility forecasting via in-context learningcited by 1
  21. 2026Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learningcited by 1
  22. 2026SERE: Structural Example Retrieval for Enhancing LLMs in Event Causality Identificationcited by 1
  23. 2026Discourse-Role Labels as Presentation-Time Variables for Context Use in Language Modelscited by 1
  24. 2026Document-Authored Control-Signal Impersonation: A Low-Cost Indirect Prompt Attack on RAG Safety Boundariescited by 1
  25. 2026When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Rerankingcited by 1
  26. 2024Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Modelscited by 0
  27. 2025Exploring Imbalanced Annotations for Effective In-Context Learningcited by 0
  28. 2025ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classificationcited by 0
  29. 2025Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translationcited by 0
  30. 2025Improving Dialogue State Tracking through Combinatorial Search for In-Context Examplescited by 0
  31. 2025Factors affecting the in-context learning abilities of LLMs for dialogue state trackingcited by 0
  32. 2025Mockingbird: How does LLM perform in general machine learning tasks?cited by 0
  33. 2025CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Modelscited by 0
  34. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspectivecited by 0
  35. 2026Geometry-Aware Hallucination Detection in Large Language Modelscited by 0
  36. 2026Love Me, Love My Label: Rethinking the Role of Labels in Prompt Retrieval for Visual In-Context Learningcited by 0
  37. 2026A Comparative Study of Demonstration Selection for Practical Large Language Models-based Next POI Predictioncited by 0
  38. 2026UCS: Estimating Unseen Coverage for Improved In-Context Learningcited by 0

50 most cited of 58 All 58 (JSON)