# Quantity vs Quality: Investigating the Trade-Off between Sample Size and Label Reliability

arXiv: https://arxiv.org/abs/2204.09462 · HTML: https://mlsanity.com/paper/2204.09462/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 8 of 8)

- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Cost-Effective Active Learning for Deep Image Classification](https://mlsanity.com/paper/1701.03551/) (2017)
- [Training Convolutional Networks with Noisy Labels](https://mlsanity.com/paper/1406.2080/) (2014)
- [An Ensemble of Simple Convolutional Neural Network Models for MNIST Digit Recognition](https://mlsanity.com/paper/2008.10400/) (2020)
- [A Comparison of Contextual and Non-Contextual Preference Ranking for Set Addition Problems](https://mlsanity.com/paper/2107.04438/) (2021)

## Abstract (arXiv)

In this paper, we study learning in probabilistic domains where the learner may receive incorrect labels but can improve the reliability of labels by repeatedly sampling them. In such a setting, one faces the problem of whether the fixed budget for obtaining training examples should rather be used for obtaining all different examples or for improving the label quality of a smaller number of examples by re-sampling their labels. We motivate this problem in an application to compare the strength of poker hands where the training signal depends on the hidden community cards, and then study it in depth in an artificial setting where we insert controlled noise levels into the MNIST database. Our results show that with increasing levels of noise, resampling previous examples becomes increasingly more important than obtaining new examples, as classifier performance deteriorates when the number of incorrect labels is too high. In addition, we propose two different validation strategies; switching from lower to higher validations over the course of training and using chi-square statistics to approximate the confidence in obtained labels.