# Predicting data value before collection: A coefficient for prioritizing sources under random distribution shift

arXiv: https://arxiv.org/abs/2504.06570 · HTML: https://mlsanity.com/paper/2504.06570/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2025.

## Built on by (most cited first, 2 of 2)

- [When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels](https://mlsanity.com/paper/2609.13148/) (2026)
- [Augmented Inverse Hybrid Weighting: Robust Inference under Deterministic and Random Distribution Shifts](https://mlsanity.com/paper/2608.00701/) (2026)

## Abstract (arXiv)

Researchers often face choices between multiple data sources that differ in quality, cost, and representativeness. Which sources will most improve predictive performance? We study this data prioritization problem under a random distribution shift model, where candidate sources arise from random perturbations to a target population. We propose the Data Usefulness Coefficient (DUC), which predicts the reduction in prediction error from adding a dataset to training, using only covariate summary statistics and no outcome data. We prove that under random shifts, covariate differences between sources are informative about outcome prediction quality. Through theory and experiments on synthetic and real data, we demonstrate that DUC-based selection outperforms alternative strategies, allowing more efficient resource allocation across heterogeneous data sources. The method provides interpretable rankings between candidate datasets and works for any data modality, including ordinal, categorical, and continuous data.