# Data Reliability Scoring

arXiv: https://arxiv.org/abs/2510.17085 · HTML: https://mlsanity.com/paper/2510.17085/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 59 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Crowdsourced Judgement Elicitation with Endogenous Proficiency](https://mlsanity.com/paper/1303.0799/) (2013)
- [Fundamentals of Task-Agnostic Data Valuation](https://mlsanity.com/paper/2208.12354/) (2022)
- [Dominantly Truthful Multi-task Peer Prediction with a Constant Number of Tasks](https://mlsanity.com/paper/1911.00272/) (2019)
- [Informational Substitutes](https://mlsanity.com/paper/1703.08636/) (2017)
- [Characteristic kernels on Hilbert spaces, Banach spaces, and on sets of measures](https://mlsanity.com/paper/2206.07588/) (2022)
- [Proper Dataset Valuation by Pointwise Mutual Information](https://mlsanity.com/paper/2405.18253/) (2024)
- [Surrogate Scoring Rules](https://mlsanity.com/paper/1802.09158/) (2018)
- [The Limits of Multi-task Peer Prediction](https://mlsanity.com/paper/2106.03176/) (2021)

## Built on by (most cited first, 2 of 2)

- [Peer-Predictive Self-Training for Language Model Reasoning](https://mlsanity.com/paper/2604.13356/) (2026)
- [Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics](https://mlsanity.com/paper/2608.01423/) (2026)

## Abstract (arXiv)

How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth-based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth-based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experimen -- a property we term experiment agnosticism. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.