mlsanityData to 7 Oct 2026

DemoShapley: Valuation of Demonstrations for In-Context Learning

arXiv 2410.07523 ↗First posted 10 Oct 2024cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) using in-context learning (ICL) excel in many tasks without task-specific fine-tuning. However, demonstration selection and ordering greatly impact ICL effectiveness. Focus on this issue, we propose DemoShapley, a Shapley-value based method that evaluates each demonstration's contribution by measuring its marginal effect across different prompt permutations. To further account for ICL's limited context windows and frequent low-shot settings, we introduce Beta-DemoShapley, a weighted extension that emphasizes the influence of smaller prompt sizes. Experiments on multiple benchmarks show that DemoShapley consistently outperforms existing influence-based selection strategies, while Beta-DemoShapley further improves performance in low-shot scenarios. Both methods also detect mislabeled data, enhance generalization to out-of-distribution tasks, and reduce demographic bias. Together, they provide a unified and robust framework for demonstration valuation in ICL.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

32 of 57 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2023Mistral 7Bcited by 3,065
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
20 more
  1. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  2. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  3. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  4. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  5. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  6. 2023Larger language models do in-context learning differentlycited by 386
  7. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  8. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  9. 2019Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithmscited by 147
  10. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  11. 2019Data Valuation using Reinforcement Learningcited by 115
  12. 2020A Distributional Framework for Data Valuationcited by 79
  13. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  14. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  15. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  16. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  17. 2023In-context Example Selection with Influencescited by 60
  18. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  19. 2013Bounding the Estimation Error of Sampling-based Shapley Value Approximationcited by 56
  20. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
  2. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  3. 2026More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffoldingcited by 0