# Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation

arXiv: https://arxiv.org/abs/2504.09341 · HTML: https://mlsanity.com/paper/2504.09341/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Zenseact Open Dataset: A large-scale and diverse multimodal dataset for autonomous driving](https://mlsanity.com/paper/2305.02008/) (2023)
- [The EuroCity Persons Dataset: A Novel Benchmark for Object Detection](https://mlsanity.com/paper/1805.07193/) (2018)
- [Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets](https://mlsanity.com/paper/2104.12690/) (2021)
- [CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://mlsanity.com/paper/2210.06812/) (2022)
- [Reward Modeling with Ordinal Feedback: Wisdom of the Crowd](https://mlsanity.com/paper/2411.12843/) (2024)
- [Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing](https://mlsanity.com/paper/1602.03481/) (2016)
- [Uncertainty Estimation for 3D Object Detection via Evidential Learning](https://mlsanity.com/paper/2410.23910/) (2024)
- [Online Policies for Efficient Volunteer Crowdsourcing](https://mlsanity.com/paper/2002.08474/) (2020)
- [How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators](https://mlsanity.com/paper/2502.06387/) (2025)
- [No Need to Sacrifice Data Quality for Quantity: Crowd-Informed Machine Annotation for Cost-Effective Understanding of Visual Data](https://mlsanity.com/paper/2409.00048/) (2024)
- [Decision-Aware Predictive Model Selection for Workforce Allocation](https://mlsanity.com/paper/2410.07932/) (2024)

## Abstract (arXiv)

High-quality data annotation is an essential but laborious and costly aspect of developing machine learning-based software. We explore the inherent tradeoff between annotation accuracy and cost by detecting and removing minority reports -- instances where annotators provide incorrect responses -- that indicate unnecessary redundancy in task assignments. We propose an approach to prune potentially redundant annotation task assignments before they are executed by estimating the likelihood of an annotator disagreeing with the majority vote for a given task. Our approach is informed by an empirical analysis over computer vision datasets annotated by a professional data annotation platform, which reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue. Simulations over these datasets show that we can reduce the number of annotations required by over 60% with a small compromise in label quality, saving approximately 6.6 days-equivalent of labor. Our approach provides annotation service platforms with a method to balance cost and dataset quality. Machine learning practitioners can tailor annotation accuracy levels according to specific application needs, thereby optimizing budget allocation while maintaining the data quality necessary for critical settings like autonomous driving technology.