mlsanityData to 14 Oct 2026

Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Quality

arXiv 2305.12811 ↗First posted 22 May 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

High-quality data is crucial for the success of machine learning, but labeling large datasets is often a time-consuming and costly process. While semi-supervised learning can help mitigate the need for labeled data, label quality remains an open issue due to ambiguity and disagreement among annotators. Thus, we use proposal-guided annotations as one option which leads to more consistency between annotators. However, proposing a label increases the probability of the annotators deciding in favor of this specific label. This introduces a bias which we can simulate and remove. We propose a new method CleverLabel for Cost-effective LabEling using Validated proposal-guidEd annotations and Repaired LABELs. CleverLabel can reduce labeling costs by up to 30.0%, while achieving a relative improvement in Kullback-Leibler divergence of up to 29.8% compared to the previous state-of-the-art on a multi-domain real-world image classification benchmark. CleverLabel offers a novel solution to the challenge of efficiently labeling large datasets while also improving the label quality.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 66 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

42 of 66 references.

References · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  4. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  5. 2019When Does Label Smoothing Help?cited by 988
  6. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  7. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  8. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  9. 2020A Survey of Deep Active Learningcited by 554
  10. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  11. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  12. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
30 more
  1. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  2. 2020Are we done with ImageNet?cited by 333
  3. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  4. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
  5. 2019Human uncertainty makes classification more robustcited by 235
  6. 2020Does label smoothing mitigate label noise?cited by 204
  7. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 118
  8. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  9. 2022When does dough become a bagel? Analyzing the remaining mistakes on ImageNetcited by 61
  10. 2020A Realistic Fish-Habitat Dataset to Evaluate Algorithms for Underwater Visual Analysiscited by 57
  11. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
  12. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  13. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  14. 2021Correlated Input-Dependent Label Noise in Large-Scale Image Classificationcited by 37
  15. 2022To Aggregate or Not? Learning with Separate Noisy Labelscited by 29
  16. 2021A Data-Centric Approach for Training Deep Neural Networks with Less Datacited by 28
  17. 2021Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interfacecited by 21
  18. 2021An Instance-Dependent Simulation Framework for Learning with Label Noisecited by 14
  19. 2019Analysis of Automatic Annotation Suggestions for Hard Discourse-Level Tasks in Expert Domainscited by 11
  20. 2020Beyond Cats and Dogs: Semi-supervised Classification of fuzzy labels with overclusteringcited by 11
  21. 2022Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusioncited by 10
  22. 2020MorphoCluster: Efficient Annotation of Plankton images by Clusteringcited by 9
  23. 2021Deep learning with self-supervision and uncertainty regularization to count fish in underwater imagescited by 9
  24. 2021A data-centric approach for improving ambiguous labels with combined semi-supervised classification and clusteringcited by 9
  25. 2021Scaling up instance annotation via label propagationcited by 9
  26. 2022The Missing Link: Finding label relations across datasetscited by 9
  27. 2022How Would The Viewer Feel? Estimating Wellbeing From Video Scenarioscited by 9
  28. 2022Beyond Hard Labels: Investigating data label distributionscited by 7
  29. 2022Faint Features Tell: Automatic Vertebrae Fracture Screening Assisted by Contrastive Learningcited by 5
  30. 2018The Impact of Annotation Guidelines and Annotated Data on Extracting App Features from App Reviewscited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  2. 2023Annotating Ambiguous Images: General Annotation Strategy for High-Quality Data with Real-World Biomedical Validationcited by 1
  3. 2025From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasetscited by 0