# Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?

arXiv: https://arxiv.org/abs/1911.07128 · HTML: https://mlsanity.com/paper/1911.07128/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2019.

## Built on by (most cited first, 9 of 9)

- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [Data Shapley Valuation for Efficient Batch Active Learning](https://mlsanity.com/paper/2104.08312/) (2021)
- [Evaluating the Values of Sources in Transfer Learning](https://mlsanity.com/paper/2104.12567/) (2021)
- [Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning](https://mlsanity.com/paper/2106.02938/) (2021)
- [Differentially Private Shapley Values for Data Evaluation](https://mlsanity.com/paper/2206.00511/) (2022)
- [Efficient Medical Image Assessment via Self-supervised Learning](https://mlsanity.com/paper/2209.14434/) (2022)
- [A Comprehensive Study of Shapley Value in Data Analytics](https://mlsanity.com/paper/2412.01460/) (2024)
- [PrivaDE: Privacy-preserving Data Evaluation for Blockchain-based Data Marketplaces](https://mlsanity.com/paper/2510.18109/) (2025)
- [A case for data valuation transparency via DValCards](https://mlsanity.com/paper/2506.23349/) (2025)

## Abstract (arXiv)

Quantifying the importance of each training point to a learning task is a fundamental problem in machine learning and the estimated importance scores have been leveraged to guide a range of data workflows such as data summarization and domain adaption. One simple idea is to use the leave-one-out error of each training point to indicate its importance. Recent work has also proposed to use the Shapley value, as it defines a unique value distribution scheme that satisfies a set of appealing properties. However, calculating Shapley values is often expensive, which limits its applicability in real-world applications at scale. Multiple heuristics to improve the scalability of calculating Shapley values have been proposed recently, with the potential risk of compromising their utility in real-world applications. textitHow well do existing data quantification methods perform on existing workflows? How do these methods compare with each other, empirically and theoretically? Must we sacrifice scalability for the utility in these workflows when using these methods? In this paper, we conduct a novel theoretical analysis comparing the utility of different importance quantification methods, and report extensive experimental studies on existing and proposed workflows such as noisy label detection, watermark removal, data summarization, data acquisition, and domain adaptation. We show that Shapley value approximation based on a KNN surrogate over pre-trained feature embeddings obtains comparable utility with existing algorithms while achieving significant scalability improvement, often by orders of magnitude. Our theoretical analysis also justifies its advantage over the leave-one-out error. The code is available at urlhttps://github.com/AI-secure/Shapley-Study.