mlsanityData to 14 Oct 2026

Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits

arXiv 2405.03875 ↗First posted 6 May 2024cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Data Shapley provides a principled approach to data valuation and plays a crucial role in data-centric machine learning (ML) research. Data selection is considered a standard application of Data Shapley. However, its data selection performance has shown to be inconsistent across settings in the literature. This study aims to deepen our understanding of this phenomenon. We introduce a hypothesis testing framework and show that Data Shapley's performance can be no better than random selection without specific constraints on utility functions. We identify a class of utility functions, monotonically transformed modular functions, within which Data Shapley optimally selects data. Based on this insight, we propose a heuristic for predicting Data Shapley's effectiveness in data selection tasks. Our experiments corroborate these findings, adding new insights into when Data Shapley may or may not succeed.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

30 of 52 references.

References · 30

  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  2. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  3. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  4. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  5. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  6. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 281
  7. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  8. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 165
  9. 2019Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithmscited by 148
  10. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 132
  11. 2021Sampling Permutations for Shapley Value Estimationcited by 102
  12. 2020A Distributional Framework for Data Valuationcited by 80
18 more
  1. 2022Impossibility Theorems for Feature Attributioncited by 78
  2. 2020Collaborative Machine Learning with Incentive-Aware Model Rewardscited by 71
  3. 2023LAVA: Data Valuation without Pre-Specified Learning Algorithmscited by 69
  4. 2023OpenDataVal: a Unified Benchmark for Data Valuationcited by 55
  5. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 53
  6. 2022Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experimentscited by 50
  7. 2021Revisiting Methods for Finding Influential Examplescited by 39
  8. 2020Efficient computation and analysis of distributional Shapley valuescited by 38
  9. 2023Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Valuecited by 38
  10. 2023Data Valuation Without Training of a Modelcited by 35
  11. 2020Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Datasetcited by 34
  12. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  13. 2021Incentivizing Collaboration in Machine Learning via Synthetic Data Rewardscited by 20
  14. 2022Fundamentals of Task-Agnostic Data Valuationcited by 19
  15. 2023Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuationcited by 19
  16. 2024Efficient Data Shapley for Weighted Nearest Neighbor Algorithmscited by 18
  17. 2023A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"cited by 11
  18. 2022CheckSel: Efficient and Accurate Data-valuation Through Online Checkpoint Selectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 10; 2026: 13.

29 papers.

Sort:

By area:

  1. 2024Data Shapley in One Training Runcited by 81
  2. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  3. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  4. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  5. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 7
  6. 2026Motion Attribution for Video Generationcited by 5
  7. 20242D-OOB: Attributing Data Contribution Through Joint Valuation Frameworkcited by 4
  8. 2026TreeGrad-Ranker: Feature Ranking via O(L)-Time Gradients for Decision Treescited by 4
  9. 2024A Comprehensive Study of Shapley Value in Data Analyticscited by 2
  10. 2026Interaction-Aware Influence Functions for Group Attributioncited by 2
  11. 2025Unifying and Optimizing Data Values for Selection via Sequential Decision-Makingcited by 1
  12. 2025On the Impact of the Utility in Semivalue-based Data Valuationcited by 1
17 more
  1. 2025How to safely discard features based on aggregate SHAP valuescited by 1
  2. 2025KAIROS: Scalable Model-Agnostic Data Valuationcited by 1
  3. 2026Online Data Selection for Instruction Tuning via Gaussian Processescited by 1
  4. 2025Backbone Augmented Training for Adaptationscited by 0
  5. 2025A case for data valuation transparency via DValCardscited by 0
  6. 2025f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomnesscited by 0
  7. 2025Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Methodcited by 0
  8. 2025Reliable and Private Utility Signaling for Data Marketscited by 0
  9. 2026SCARV: Structure-Constrained Aggregation for Stable Sample Ranking in Redundant NLP Datasetscited by 0
  10. 2026Let the Target Select for Itself: Data Selection via Target-Aligned Pathscited by 0
  11. 2026Is Data Shapley Not Better than Random in Data Selection? Ask NASHcited by 0
  12. 2026Constraint-Data-Value-Maximization: Utilizing Data Attribution for Effective Data Pruning in Low-Data Environmentscited by 0
  13. 2026On the Fragility of Data Attribution When Learning Is Distributedcited by 0
  14. 2026Nonlinear Axiomatic Attribution for Cooperative Gamescited by 0
  15. 2026Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysiscited by 0
  16. 2026Marginal Log-Likelihood Increments under Dirichlet-Smoothed Markov Estimationcited by 0
  17. 2026GraphSelect for Budgeted Representation Selection in Multimodal Graph Inferencecited by 0