# Don't Waste a Single Annotation: Improving Single-Label Classifiers Through Soft Labels

arXiv: https://arxiv.org/abs/2311.05265 · HTML: https://mlsanity.com/paper/2311.05265/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 14 of 14)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- ['Less Than One'-Shot Learning: Learning N Classes From M<N Samples](https://mlsanity.com/paper/2009.08449/) (2020)
- [VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on Twitter](https://mlsanity.com/paper/2301.06660/) (2023)
- [Crowdsourcing Semantic Label Propagation in Relation Classification](https://mlsanity.com/paper/1809.00537/) (2018)

## Built on by (most cited first, 17 of 17)

- [A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice](https://mlsanity.com/paper/2404.16958/) (2024)
- [Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?](https://mlsanity.com/paper/2408.14141/) (2024)
- [UKElectionNarratives: A Dataset of Misleading Narratives Surrounding Recent UK General Elections](https://mlsanity.com/paper/2505.05459/) (2025)
- [RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science](https://mlsanity.com/paper/2408.08217/) (2024)
- [Distributions In, Distributions Out: The Case for Soft-Label Training](https://mlsanity.com/paper/2511.14117/) (2025)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media](https://mlsanity.com/paper/2410.14515/) (2024)
- [A Dataset for Analysing News Framing in Chinese Media](https://mlsanity.com/paper/2503.04439/) (2025)
- [Efficient Annotator Reliability Assessment with EffiARA](https://mlsanity.com/paper/2504.00589/) (2025)
- [SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation](https://mlsanity.com/paper/2603.15150/) (2026)
- [SCRum-9: Multilingual Stance Classification over Rumours on Social Media](https://mlsanity.com/paper/2505.18916/) (2025)
- [Learning from Hard Labels with Additional Supervision on Non-Hard-Labeled Classes](https://mlsanity.com/paper/2507.18098/) (2025)
- [ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning](https://mlsanity.com/paper/2602.12714/) (2026)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Metric-Dependent Annotation Saturation for Learning from Label Distributions](https://mlsanity.com/paper/2605.29797/) (2026)
- [Reliability-Aware Sexism Detection: Combining DPO with Annotator Agreement and Token-Level Confidence Scoring](https://mlsanity.com/paper/2608.12330/) (2026)
- [Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection](https://mlsanity.com/paper/2609.15608/) (2026)

## Abstract (arXiv)

In this paper, we address the limitations of the common data annotation and training methods for objective single-label classification tasks. Typically, when annotating such tasks annotators are only asked to provide a single label for each sample and annotator disagreement is discarded when a final hard label is decided through majority voting. We challenge this traditional approach, acknowledging that determining the appropriate label can be difficult due to the ambiguity and lack of context in the data samples. Rather than discarding the information from such ambiguous annotations, our soft label method makes use of them for training. Our findings indicate that additional annotator information, such as confidence, secondary label and disagreement, can be used to effectively generate soft labels. Training classifiers with these soft labels then leads to improved performance and calibration on the hard label test set.