# OpenDataVal: a Unified Benchmark for Data Valuation

arXiv: https://arxiv.org/abs/2306.10577 · HTML: https://mlsanity.com/paper/2306.10577/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 49 references that are in the map.
- First posted on arXiv 18 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2110.14049/) (2021)
- [Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms](https://mlsanity.com/paper/1908.08619/) (2019)
- [Data Banzhaf: A Robust Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2205.15466/) (2022)
- [Data Valuation using Reinforcement Learning](https://mlsanity.com/paper/1909.11671/) (2019)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts](https://mlsanity.com/paper/2202.06523/) (2022)
- [A Marketplace for Data: An Algorithmic Solution](https://mlsanity.com/paper/1805.08125/) (2018)
- [Collaborative Machine Learning with Incentive-Aware Model Rewards](https://mlsanity.com/paper/2010.12797/) (2020)
- [LAVA: Data Valuation without Pre-Specified Learning Algorithms](https://mlsanity.com/paper/2305.00054/) (2023)
- [Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments](https://mlsanity.com/paper/2206.10013/) (2022)
- [Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value](https://mlsanity.com/paper/2304.07718/) (2023)
- [Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Dataset](https://mlsanity.com/paper/2010.08006/) (2020)
- [Revisiting the Gelman-Rubin Diagnostic](https://mlsanity.com/paper/1812.09384/) (2018)
- [Private Data Valuation and Fair Payment in Data Marketplaces](https://mlsanity.com/paper/2210.08723/) (2022)
- [Accuracy on the Curve: On the Nonlinear Correlation of ML Performance Between Data Subpopulations](https://mlsanity.com/paper/2305.02995/) (2023)

## Built on by (most cited first, 30 of 55)

- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [Data Shapley in One Training Run](https://mlsanity.com/paper/2406.11011/) (2024)
- [shapiq: Shapley Interactions for Machine Learning](https://mlsanity.com/paper/2410.01649/) (2024)
- [Rethinking Data Shapley for Data Selection Tasks: Misleads and Merits](https://mlsanity.com/paper/2405.03875/) (2024)
- [Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attribution](https://mlsanity.com/paper/2401.15866/) (2024)
- [textttdattri: A Library for Efficient Data Attribution](https://mlsanity.com/paper/2410.04555/) (2024)
- [Data Valuation and Detections in Federated Learning](https://mlsanity.com/paper/2311.05304/) (2023)
- [Scaling Laws for the Value of Individual Data Points in Machine Learning](https://mlsanity.com/paper/2405.20456/) (2024)
- [Data Acquisition: A New Frontier in Data-centric AI](https://mlsanity.com/paper/2311.13712/) (2023)
- [Matchmaker: Self-Improving Large Language Model Programs for Schema Matching](https://mlsanity.com/paper/2410.24105/) (2024)
- [What Data Enables Optimal Decisions? An Exact Characterization for Linear Optimization](https://mlsanity.com/paper/2505.21692/) (2025)
- [Influence Guided Context Selection for Effective Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.21359/) (2025)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [DAVED: Data Acquisition via Experimental Design for Data Markets](https://mlsanity.com/paper/2403.13893/) (2024)
- [TimeInf: Time Series Data Contribution via Influence Functions](https://mlsanity.com/paper/2407.15247/) (2024)
- [To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis](https://mlsanity.com/paper/2512.05925/) (2025)
- [Self-Tuning Self-Supervised Image Anomaly Detection](https://mlsanity.com/paper/2306.12033/) (2023)
- [On the Inflation of KNN-Shapley Value](https://mlsanity.com/paper/2405.17489/) (2024)
- [CHG Shapley: Efficient Data Valuation and Selection towards Trustworthy Machine Learning](https://mlsanity.com/paper/2406.11730/) (2024)
- [2D-OOB: Attributing Data Contribution Through Joint Valuation Framework](https://mlsanity.com/paper/2408.03572/) (2024)
- [Targeted synthetic data generation for tabular data via hardness characterization](https://mlsanity.com/paper/2410.00759/) (2024)
- [Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining](https://mlsanity.com/paper/2505.24261/) (2025)
- [Uncertainty-aware Evaluation of Auxiliary Anomalies with the Expected Anomaly Posterior](https://mlsanity.com/paper/2405.13699/) (2024)
- [Data Measurements for Decentralized Data Markets](https://mlsanity.com/paper/2406.04257/) (2024)
- [Adversarial Attacks on Data Attribution](https://mlsanity.com/paper/2409.05657/) (2024)
- [What is the Right Notion of Distance between Predict-then-Optimize Tasks?](https://mlsanity.com/paper/2409.06997/) (2024)
- [SAVA: Scalable Learning-Agnostic Data Valuation](https://mlsanity.com/paper/2406.01130/) (2024)
- [Is Data Valuation Learnable and Interpretable?](https://mlsanity.com/paper/2406.02612/) (2024)
- [LossVal: Efficient Data Valuation for Neural Networks](https://mlsanity.com/paper/2412.04158/) (2024)
- [Unifying and Optimizing Data Values for Selection via Sequential Decision-Making](https://mlsanity.com/paper/2502.04554/) (2025)

## Abstract (arXiv)

Assessing the quality and impact of individual data points is critical for improving model performance and mitigating undesirable biases within the training dataset. Several data valuation algorithms have been proposed to quantify data quality, however, there lacks a systemic and standardized benchmarking system for data valuation. In this paper, we introduce OpenDataVal, an easy-to-use and unified benchmark framework that empowers researchers and practitioners to apply and compare various data valuation algorithms. OpenDataVal provides an integrated environment that includes (i) a diverse collection of image, natural language, and tabular datasets, (ii) implementations of eleven different state-of-the-art data valuation algorithms, and (iii) a prediction model API that can import any models in scikit-learn. Furthermore, we propose four downstream machine learning tasks for evaluating the quality of data values. We perform benchmarking analysis using OpenDataVal, quantifying and comparing the efficacy of state-of-the-art data valuation approaches. We find that no single algorithm performs uniformly best across all tasks, and an appropriate algorithm should be employed for a user's downstream task. OpenDataVal is publicly available at https://opendataval.github.io with comprehensive documentation. Furthermore, we provide a leaderboard where researchers can evaluate the effectiveness of their own data valuation algorithms.