# Trustable Co-label Learning from Multiple Noisy Annotators

arXiv: https://arxiv.org/abs/2203.04199 · HTML: https://mlsanity.com/paper/2203.04199/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 98 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Joint Optimization Framework for Learning with Noisy Labels](https://mlsanity.com/paper/1803.11364/) (2018)
- [WebVision Database: Visual Learning and Understanding from Web Data](https://mlsanity.com/paper/1708.02862/) (2017)
- [Are Anchor Points Really Indispensable in Label-Noise Learning?](https://mlsanity.com/paper/1906.00189/) (2019)
- [Learning from Noisy Labels with Distillation](https://mlsanity.com/paper/1703.02391/) (2017)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [A Second-Order Approach to Learning with Instance-Dependent Label Noise](https://mlsanity.com/paper/2012.11854/) (2020)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Learning with Bounded Instance- and Label-dependent Label Noise](https://mlsanity.com/paper/1709.03768/) (2017)
- [Learning the Structure of Generative Models without Labeled Data](https://mlsanity.com/paper/1703.00854/) (2017)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)
- [Learning Dependency Structures for Weak Supervision Models](https://mlsanity.com/paper/1903.05844/) (2019)
- [Learning from Crowds by Modeling Common Confusions](https://mlsanity.com/paper/2012.13052/) (2020)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Exploiting Web Images for Dataset Construction: A Domain Robust Approach](https://mlsanity.com/paper/1611.07156/) (2016)
- [Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing](https://mlsanity.com/paper/1411.4086/) (2014)
- [Learning Supervised Topic Models for Classification and Regression from Crowds](https://mlsanity.com/paper/1808.05902/) (2018)

## Built on by (most cited first, 12 of 12)

- [M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy](https://mlsanity.com/paper/2312.15927/) (2023)
- [DANCE: Dual-View Distribution Alignment for Dataset Condensation](https://mlsanity.com/paper/2406.01063/) (2024)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [Coupled Confusion Correction: Learning from Crowds with Sparse Annotations](https://mlsanity.com/paper/2312.07331/) (2023)
- [Effective and Robust Adversarial Training against Data and Label Corruptions](https://mlsanity.com/paper/2405.04191/) (2024)
- [L2CU: Learning to Complement Unseen Users](https://mlsanity.com/paper/2601.06119/) (2026)
- [Multi-Dimensional Refinement Graph Convolutional Network with Robust Decouple Loss for Fine-Grained Skeleton-Based Action Recognition](https://mlsanity.com/paper/2306.15321/) (2023)
- [Multi-Label Noise Transition Matrix Estimation with Label Correlations: Theory and Algorithm](https://mlsanity.com/paper/2309.12706/) (2023)
- [A Unified Optimal Transport Framework for Cross-Modal Retrieval with Noisy Labels](https://mlsanity.com/paper/2403.13480/) (2024)
- [crowd-hpo: Realistic Hyperparameter Optimization and Benchmarking for Learning from Crowds with Noisy Labels](https://mlsanity.com/paper/2504.09085/) (2025)
- [Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval](https://mlsanity.com/paper/2512.24064/) (2025)
- [Privacy-Preserving Model Transcription with Differentially Private Synthetic Distillation](https://mlsanity.com/paper/2601.19090/) (2026)

## Abstract (arXiv)

Supervised deep learning depends on massive accurately annotated examples, which is usually impractical in many real-world scenarios. A typical alternative is learning from multiple noisy annotators. Numerous earlier works assume that all labels are noisy, while it is usually the case that a few trusted samples with clean labels are available. This raises the following important question: how can we effectively use a small amount of trusted data to facilitate robust classifier learning from multiple annotators? This paper proposes a data-efficient approach, called Trustable Co-label Learning (TCL), to learn deep classifiers from multiple noisy annotators when a small set of trusted data is available. This approach follows the coupled-view learning manner, which jointly learns the data classifier and the label aggregator. It effectively uses trusted data as a guide to generate trustable soft labels (termed co-labels). A co-label learning can then be performed by alternately reannotating the pseudo labels and refining the classifiers. In addition, we further improve TCL for a special complete data case, where each instance is labeled by all annotators and the label aggregator is represented by multilayer neural networks to enhance model capacity. Extensive experiments on synthetic and real datasets clearly demonstrate the effectiveness and robustness of the proposed approach. Source code is available at https://github.com/ShikunLi/TCL