mlsanityData to 14 Oct 2026

To Aggregate or Not? Learning with Separate Noisy Labels

arXiv 2206.07181 ↗First posted 14 Jun 2022cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The rawly collected training data often comes with separate noisy labels collected from multiple imperfect annotators (e.g., via crowdsourcing). A typical way of using these separate labels is to first aggregate them into one and apply standard training methods. The literature has also studied extensively on effective aggregation approaches. This paper revisits this choice and aims to provide an answer to the question of whether one should aggregate separate noisy labels into single ones or use them separately as given. We theoretically analyze the performance of both approaches under the empirical risk minimization framework for a number of popular loss functions, including the ones designed specifically for the problem of learning with noisy labels. Our theorems conclude that label separation is preferred over label aggregation when the noise rates are high, or the number of labelers/annotations is insufficient. Extensive empirical results validate our conclusions.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 49 of its 93 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

49 of 93 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  3. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  4. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  5. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 500
  6. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  7. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  8. 2019How does Disagreement Help Generalization against Label Corruption?cited by 390
  9. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  10. 2020Combating noisy labels by agreement: A joint training method with co-regularizationcited by 289
  11. 2020Normalized Loss Functions for Deep Learning with Noisy Labelscited by 257
  12. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
37 more
  1. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  2. 2019Human uncertainty makes classification more robustcited by 235
  3. 2019Are Anchor Points Really Indispensable in Label-Noise Learning?cited by 208
  4. 2020Does label smoothing mitigate label noise?cited by 204
  5. 2016Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challengecited by 201
  6. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  7. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  8. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  9. 2020A Survey of Label-noise Representation Learning: Past, Present and Futurecited by 122
  10. 2018Training Complex Models with Multi-Task Weak Supervisioncited by 106
  11. 2017Deep learning from crowdscited by 105
  12. 2020The Emerging Trends of Multi-Label Learningcited by 100
  13. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  14. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 96
  15. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  16. 2020A Second-Order Approach to Learning with Instance-Dependent Label Noisecited by 90
  17. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  18. 2017Who Said What: Modeling Individual Labelers Improves Classificationcited by 77
  19. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  20. 2021Clusterability as an Alternative to Anchor Points When Learning with Noisy Labelscited by 65
  21. 2021Adaptive Early-Learning Correction for Segmentation from Noisy Annotationscited by 62
  22. 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
  23. 2021Open-set Label Noise Can Improve Robustness Against Inherent Label Noisecited by 59
  24. 2021Detecting Corrupted Labels Without Training a Model to Predictcited by 58
  25. 2020When Optimizing f-divergence is Robust with Label Noisecited by 44
  26. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  27. 2022Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Featurescited by 27
  28. 2021Mitigating Memorization of Noisy Labels via Regularization between Representationscited by 26
  29. 2021Understanding Instance-Level Label Noise: Disparate Impacts and Treatmentscited by 22
  30. 2018Learning Supervised Topic Models for Classification and Regression from Crowdscited by 16
  31. 2021Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyondcited by 16
  32. 2020Policy Learning Using Weak Supervisioncited by 13
  33. 2021Constrained Instance and Class Reweighting for Robust Learning under Label Noisecited by 12
  34. 2020SemiNLL: A Framework of Noisy-Label Learning by Semi-Supervised Learningcited by 11
  35. 2021Multiplicative Reweighting for Robust Neural Network Optimizationcited by 4
  36. 2022GearNet: Stepwise Dual Learning for Weakly Supervised Domain Adaptationcited by 4
  37. 2021Alleviating Noisy-label Effects in Image Classification via Probability Transition Matrixcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 6; 2024: 10; 2025: 8.

29 papers.

Sort:

By area:

  1. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 75
  2. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  3. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
  4. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  5. 2025Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Modelscited by 28
  6. 2023Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Modelscited by 26
  7. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  8. 2023Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowdscited by 11
  9. 2023Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurationscited by 9
  10. 2022Human-in-the-Loop Mixupcited by 8
  11. 2023The Importance of Human-Labeled Data in the Era of LLMscited by 8
  12. 2024FedFixer: Mitigating Heterogeneous Label Noise in Federated Learningcited by 8
17 more
  1. 2024Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budgetcited by 7
  2. 2024Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedbackcited by 6
  3. 2025When VLMs Meet Image Classification: Test Sets Renovation via Missing Label Identificationcited by 5
  4. 2025SelectMix: Enhancing Label Noise Robustness through Targeted Sample Mixingcited by 5
  5. 2024Visual Objectification in Films: Towards a New AI Task for Video Interpretationcited by 4
  6. 2022On the Ramifications of Human Label Uncertaintycited by 3
  7. 2023Label Smarter, Not Harder: CleverLabel for Faster Annotation of Ambiguous Image Classification with Higher Qualitycited by 3
  8. 2025Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrestcited by 3
  9. 2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
  10. 2024Perceptual Quality-based Model Training under Annotator Label Uncertaintycited by 2
  11. 2024Bayesian Detector Combination for Object Detection with Crowdsourced Annotationscited by 2
  12. 2024Hands-On Tutorial: Labeling with LLM and Human-in-the-Loopcited by 2
  13. 2025Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregationcited by 2
  14. 2023Handwritten Text Recognition from Crowdsourced Annotationscited by 1
  15. 2024Exploring the Influence of Label Aggregation on Minority Voices: Implications for Dataset Bias and Model Trainingcited by 1
  16. 2025MObyGaze: a film dataset of multimodal objectification densely annotated by expertscited by 1
  17. 2025The Majority Vote Paradigm Shift: When Popular Meets Optimalcited by 0