# Rethinking Crowdsourcing Annotation: Partial Annotation with Salient Labels for Multi-Label Image Classification

arXiv: https://arxiv.org/abs/2109.02688 · HTML: https://mlsanity.com/paper/2109.02688/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 73 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Learning Visual Features from Large Weakly Supervised Data](https://mlsanity.com/paper/1511.02251/) (2015)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Multi-Label Classification with Label Graph Superimposing](https://mlsanity.com/paper/1911.09243/) (2019)
- [Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels](https://mlsanity.com/paper/1512.06974/) (2015)
- [Improving Pairwise Ranking for Multi-label Image Classification](https://mlsanity.com/paper/1704.03135/) (2017)
- [Relation Network for Multi-label Aerial Image Classification](https://mlsanity.com/paper/1907.07274/) (2019)
- [Recover Subjective Quality Scores from Noisy Measurements](https://mlsanity.com/paper/1611.01715/) (2016)
- [Inferring ground truth from multi-annotator ordinal data: a probabilistic approach](https://mlsanity.com/paper/1305.0015/) (2013)
- [Lake Ice Monitoring with Webcams and Crowd-Sourced Images](https://mlsanity.com/paper/2002.07875/) (2020)
- [SCIDA: Self-Correction Integrated Domain Adaptation from Single- to Multi-label Aerial Images](https://mlsanity.com/paper/2108.06810/) (2021)

## Built on by (most cited first, 7 of 7)

- [Multi-label Classification with Partial Annotations using Class-aware Selective Loss](https://mlsanity.com/paper/2110.10955/) (2021)
- [Deep Active Learning for Multi-Label Classification of Remote Sensing Images](https://mlsanity.com/paper/2212.01165/) (2022)
- [Category-wise Fine-Tuning: Resisting Incorrect Pseudo-Labels in Multi-Label Image Classification with Partial Labels](https://mlsanity.com/paper/2401.16991/) (2024)
- [Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelliAgent](https://mlsanity.com/paper/2511.23436/) (2025)
- [A Survey on Incomplete Multi-label Learning: Recent Advances and Future Trends](https://mlsanity.com/paper/2406.06119/) (2024)
- [CLIP the Landscape: Automated Tagging of Crowdsourced Landscape Images](https://mlsanity.com/paper/2506.12214/) (2025)
- [Data-Centric Benchmark for Label Noise Estimation and Ranking in Remote Sensing Binary Building Segmentation](https://mlsanity.com/paper/2603.00604/) (2026)

## Abstract (arXiv)

Annotated images are required for both supervised model training and evaluation in image classification. Manually annotating images is arduous and expensive, especially for multi-labeled images. A recent trend for conducting such laboursome annotation tasks is through crowdsourcing, where images are annotated by volunteers or paid workers online (e.g., workers of Amazon Mechanical Turk) from scratch. However, the quality of crowdsourcing image annotations cannot be guaranteed, and incompleteness and incorrectness are two major concerns for crowdsourcing annotations. To address such concerns, we have a rethinking of crowdsourcing annotations: Our simple hypothesis is that if the annotators only partially annotate multi-label images with salient labels they are confident in, there will be fewer annotation errors and annotators will spend less time on uncertain labels. As a pleasant surprise, with the same annotation budget, we show a multi-label image classifier supervised by images with salient annotations can outperform models supervised by fully annotated images. Our method contributions are 2-fold: An active learning way is proposed to acquire salient labels for multi-label images; and a novel Adaptive Temperature Associated Model (ATAM) specifically using partial annotations is proposed for multi-label image classification. We conduct experiments on practical crowdsourcing data, the Open Street Map (OSM) dataset and benchmark dataset COCO 2014. When compared with state-of-the-art classification methods trained on fully annotated images, the proposed ATAM can achieve higher accuracy. The proposed idea is promising for crowdsourcing data annotation. Our code will be publicly available.