# Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits

arXiv: https://arxiv.org/abs/2405.03875 · HTML: https://mlsanity.com/paper/2405.03875/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 30 of 30)

- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2110.14049/) (2021)
- [Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms](https://mlsanity.com/paper/1908.08619/) (2019)
- [Data Banzhaf: A Robust Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2205.15466/) (2022)
- [Sampling Permutations for Shapley Value Estimation](https://mlsanity.com/paper/2104.12199/) (2021)
- [A Distributional Framework for Data Valuation](https://mlsanity.com/paper/2002.12334/) (2020)
- [Impossibility Theorems for Feature Attribution](https://mlsanity.com/paper/2212.11870/) (2022)
- [Collaborative Machine Learning with Incentive-Aware Model Rewards](https://mlsanity.com/paper/2010.12797/) (2020)
- [LAVA: Data Valuation without Pre-Specified Learning Algorithms](https://mlsanity.com/paper/2305.00054/) (2023)
- [OpenDataVal: a Unified Benchmark for Data Valuation](https://mlsanity.com/paper/2306.10577/) (2023)
- [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023)
- [Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments](https://mlsanity.com/paper/2206.10013/) (2022)
- [Revisiting Methods for Finding Influential Examples](https://mlsanity.com/paper/2111.04683/) (2021)
- [Efficient computation and analysis of distributional Shapley values](https://mlsanity.com/paper/2007.01357/) (2020)
- [Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value](https://mlsanity.com/paper/2304.07718/) (2023)
- [Data Valuation Without Training of a Model](https://mlsanity.com/paper/2301.00930/) (2023)
- [Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Dataset](https://mlsanity.com/paper/2010.08006/) (2020)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [Incentivizing Collaboration in Machine Learning via Synthetic Data Rewards](https://mlsanity.com/paper/2112.09327/) (2021)
- [Fundamentals of Task-Agnostic Data Valuation](https://mlsanity.com/paper/2208.12354/) (2022)
- [Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuation](https://mlsanity.com/paper/2308.15709/) (2023)
- [Efficient Data Shapley for Weighted Nearest Neighbor Algorithms](https://mlsanity.com/paper/2401.11103/) (2024)
- [A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"](https://mlsanity.com/paper/2304.04258/) (2023)
- [CheckSel: Efficient and Accurate Data-valuation Through Online Checkpoint Selection](https://mlsanity.com/paper/2203.06814/) (2022)

## Built on by (most cited first, 29 of 29)

- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws](https://mlsanity.com/paper/2410.11820/) (2024)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [textttdattri: A Library for Efficient Data Attribution](https://mlsanity.com/paper/2410.04555/) (2024)
- [Influence Guided Context Selection for Effective Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.21359/) (2025)
- [Motion Attribution for Video Generation](https://mlsanity.com/paper/2601.08828/) (2026)
- [2D-OOB: Attributing Data Contribution Through Joint Valuation Framework](https://mlsanity.com/paper/2408.03572/) (2024)
- [TreeGrad-Ranker: Feature Ranking via O(L)-Time Gradients for Decision Trees](https://mlsanity.com/paper/2602.11623/) (2026)
- [A Comprehensive Study of Shapley Value in Data Analytics](https://mlsanity.com/paper/2412.01460/) (2024)
- [Interaction-Aware Influence Functions for Group Attribution](https://mlsanity.com/paper/2605.15675/) (2026)
- [Unifying and Optimizing Data Values for Selection via Sequential Decision-Making](https://mlsanity.com/paper/2502.04554/) (2025)
- [On the Impact of the Utility in Semivalue-based Data Valuation](https://mlsanity.com/paper/2502.06574/) (2025)
- [How to safely discard features based on aggregate SHAP values](https://mlsanity.com/paper/2503.23111/) (2025)
- [KAIROS: Scalable Model-Agnostic Data Valuation](https://mlsanity.com/paper/2506.23799/) (2025)
- [Online Data Selection for Instruction Tuning via Gaussian Processes](https://mlsanity.com/paper/2606.30077/) (2026)
- [Backbone Augmented Training for Adaptations](https://mlsanity.com/paper/2506.04288/) (2025)
- [A case for data valuation transparency via DValCards](https://mlsanity.com/paper/2506.23349/) (2025)
- [f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomness](https://mlsanity.com/paper/2510.10510/) (2025)
- [Adaptive Data Selection for Multi-Layer Perceptron Training: A Sub-linear Value-Driven Method](https://mlsanity.com/paper/2510.21286/) (2025)
- [Reliable and Private Utility Signaling for Data Markets](https://mlsanity.com/paper/2511.07975/) (2025)
- [SCARV: Structure-Constrained Aggregation for Stable Sample Ranking in Redundant NLP Datasets](https://mlsanity.com/paper/2605.00944/) (2026)
- [Let the Target Select for Itself: Data Selection via Target-Aligned Paths](https://mlsanity.com/paper/2605.09404/) (2026)
- [Is Data Shapley Not Better than Random in Data Selection? Ask NASH](https://mlsanity.com/paper/2605.10684/) (2026)
- [Constraint-Data-Value-Maximization: Utilizing Data Attribution for Effective Data Pruning in Low-Data Environments](https://mlsanity.com/paper/2605.11312/) (2026)
- [On the Fragility of Data Attribution When Learning Is Distributed](https://mlsanity.com/paper/2605.15520/) (2026)
- [Nonlinear Axiomatic Attribution for Cooperative Games](https://mlsanity.com/paper/2607.09869/) (2026)
- [Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis](https://mlsanity.com/paper/2607.15884/) (2026)
- [Marginal Log-Likelihood Increments under Dirichlet-Smoothed Markov Estimation](https://mlsanity.com/paper/2609.25675/) (2026)
- [GraphSelect for Budgeted Representation Selection in Multimodal Graph Inference](https://mlsanity.com/paper/2609.33561/) (2026)

## Abstract (arXiv)

Data Shapley provides a principled approach to data valuation and plays a crucial role in data-centric machine learning (ML) research. Data selection is considered a standard application of Data Shapley. However, its data selection performance has shown to be inconsistent across settings in the literature. This study aims to deepen our understanding of this phenomenon. We introduce a hypothesis testing framework and show that Data Shapley's performance can be no better than random selection without specific constraints on utility functions. We identify a class of utility functions, monotonically transformed modular functions, within which Data Shapley optimally selects data. Based on this insight, we propose a heuristic for predicting Data Shapley's effectiveness in data selection tasks. Our experiments corroborate these findings, adding new insights into when Data Shapley may or may not succeed.