# On Efficient and Statistical Quality Estimation for Data Annotation

arXiv: https://arxiv.org/abs/2405.11919 · HTML: https://mlsanity.com/paper/2405.11919/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 43 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 7 of 7)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://mlsanity.com/paper/2205.04596/) (2022)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://mlsanity.com/paper/2206.02280/) (2022)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Sentiment analysis is not solved! Assessing and probing sentiment classification](https://mlsanity.com/paper/1906.05887/) (2019)

## Built on by (most cited first, 8 of 8)

- [When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA](https://mlsanity.com/paper/2510.04849/) (2025)
- [When Incentives Backfire, Data Stops Being Human](https://mlsanity.com/paper/2502.07732/) (2025)
- [Trust & Safety of LLMs and LLMs in Trust & Safety](https://mlsanity.com/paper/2412.02113/) (2024)
- [Incentivizing High-Quality Human Annotations with Golden Questions](https://mlsanity.com/paper/2505.19134/) (2025)
- [Users as Annotators: LLM Preference Learning from Comparison Mode](https://mlsanity.com/paper/2510.13830/) (2025)
- [DanceCrafter: Fine-Grained Text-Driven Controllable Dance Generation via Choreographic Syntax](https://mlsanity.com/paper/2604.18648/) (2026)
- [Looks Can be Deceiving: Annotator and Reviewer Performance Across Imagery Sources in Crowd-Sourced Aerial Damage Assessment](https://mlsanity.com/paper/2608.14942/) (2026)
- [LLM Judge Validation Under Sparse Overlap: From Inference to Design](https://mlsanity.com/paper/2609.31857/) (2026)

## Abstract (arXiv)

Annotated datasets are an essential ingredient to train, evaluate, compare and productionalize supervised machine learning models. It is therefore imperative that annotations are of high quality. For their creation, good quality management and thereby reliable quality estimates are needed. Then, if quality is insufficient during the annotation process, rectifying measures can be taken to improve it. Quality estimation is often performed by having experts manually label instances as correct or incorrect. But checking all annotated instances tends to be expensive. Therefore, in practice, usually only subsets are inspected; sizes are chosen mostly without justification or regard to statistical power and more often than not, are relatively small. Basing estimates on small sample sizes, however, can lead to imprecise values for the error rate. Using unnecessarily large sample sizes costs money that could be better spent, for instance on more annotations. Therefore, we first describe in detail how to use confidence intervals for finding the minimal sample size needed to estimate the annotation error rate. Then, we propose applying acceptance sampling as an alternative to error rate estimation We show that acceptance sampling can reduce the required sample sizes up to 50% while providing the same statistical guarantees.