mlsanityData to 7 Oct 2026

In-Context Demonstration Selection with Cross Entropy Difference

arXiv 2305.14726 ↗First posted 24 May 2023cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. However, selecting the best in-context examples is challenging because model performance can vary widely depending on the selected examples. We present a cross-entropy difference (CED) method for selecting in-context demonstrations. Our method is based on the observation that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example by a language model that was finetuned on that demonstration. We utilize parameter efficient finetuning to train small models on training data that are used for computing the cross-entropy difference between a test example and every candidate in-context demonstration. This metric is used to rank and select in-context demonstrations independently for each test input. We evaluate our method on a mix-domain dataset that combines 8 benchmarks, representing 4 text generation tasks, showing that CED for in-context demonstration selection can improve performance for a variety of LLMs.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

28 of 37 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
16 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  6. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  7. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  8. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  9. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  10. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  11. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  12. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  13. 2019Learning a Multi-Domain Curriculum for Neural Machine Translationcited by 16
  14. 2021The Trade-offs of Domain Adaptation for Neural Language Modelscited by 14
  15. 2023How Does In-Context Learning Help Prompt Tuning?cited by 11
  16. 2021On the Complementarity of Data Selection and Fine Tuning for Domain Adaptationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 6; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  2. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  3. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  4. 2023Generative Calibration for In-context Learningcited by 20
  5. 2024Monotonic Paraphrasing Improves Generalization of Language Model Promptingcited by 10
  6. 2024FamiCom: Further Demystifying Prompts for Language Models with Task-Agnostic Performance Estimationcited by 8
  7. 2024Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalitiescited by 7
  8. 2023Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Modelscited by 6
  9. 2025Topic Coverage-based Demonstration Retrieval for In-Context Learningcited by 3
  10. 2025Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detectioncited by 2
  11. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  12. 2024Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generationcited by 1
5 more
  1. 2025D.Va: Validate Your Demonstration First Before You Use Itcited by 1
  2. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  3. 2026Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selectioncited by 1
  4. 2025Lightweight Time Series Data Valuation on Time Series Foundation Models via In-Context Finetuningcited by 0
  5. 2026You Only Edit Once: Incentivizing In-Context Capability of LLMs via Local Demonstration Refinementcited by 0