mlsanityData to 14 Oct 2026

Prompt Valuation Based on Shapley Values

arXiv 2312.15395 ↗First posted 24 Dec 2023cs.CL cs.DB cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel on new tasks without additional training, simply by providing natural language prompts that demonstrate how the task should be performed. Prompt ensemble methods comprehensively harness the knowledge of LLMs while mitigating individual biases and errors and further enhancing performance. However, more prompts do not necessarily lead to better results, and not all prompts are beneficial. A small number of high-quality prompts often outperform many low-quality prompts. Currently, there is a lack of a suitable method for evaluating the impact of prompts on the results. In this paper, we utilize the Shapley value to fairly quantify the contributions of prompts, helping to identify beneficial or detrimental prompts, and potentially guiding prompt valuation in data markets. Through extensive experiments employing various ensemble methods and utility functions on diverse tasks, we validate the effectiveness of using the Shapley value method for prompts as it effectively distinguishes and quantifies the contributions of each prompt.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 24 Dec 2023.

Builds on

43 of 60 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
31 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2019How Can We Know What Language Models Know?cited by 959
  11. 2021Robust fine-tuning of zero-shot modelscited by 797
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  13. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  14. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  15. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  16. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  17. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  18. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  19. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  20. 2021WARP: Word-level Adversarial ReProgrammingcited by 221
  21. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  22. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  23. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  24. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  25. 2019Leveraging BERT for Extractive Text Summarization on Lecturescited by 63
  26. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  27. 2021Shapley values for feature selection: The good, the bad, and the axiomscited by 54
  28. 2021Evaluating Gender Bias in Natural Language Inferencecited by 27
  29. 2023Improving Probability-based Prompt Selection Through Unified Evaluation and Analysiscited by 13
  30. 2023InstructEval: Systematic Evaluation of Instruction Selection Methodscited by 13
  31. 2019Pretrained Language Models for Document-Level Neural Machine Translationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 4.

10 papers.

  1. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  2. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  3. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 16
  4. 2024DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learningcited by 5
  5. 2025Efficient Leave-one-out Approximation in LLM Multi-agent Debate Based on Introspectioncited by 4
  6. 2024Data-Centric AI in the Age of Large Language Modelscited by 2
  7. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  8. 2026A Regression Framework for Understanding Prompt Component Impact on LLM Performancecited by 1
  9. 2026ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Trainingcited by 1
  10. 2026Towards Feedback-to-Plan Decisions for Self-Evolving LLM Agents in CUDA Kernel Generationcited by 0