# Learning from Multiple Annotators by Incorporating Instance Features

arXiv: https://arxiv.org/abs/2106.15146 · HTML: https://mlsanity.com/paper/2106.15146/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 10 of 10)

- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Learning with Feature-Dependent Label Noise: A Progressive Approach](https://mlsanity.com/paper/2103.07756/) (2021)
- [Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise](https://mlsanity.com/paper/2012.05458/) (2020)
- [Learning From Noisy Singly-labeled Data](https://mlsanity.com/paper/1712.04577/) (2017)
- [Learning from Crowds by Modeling Common Confusions](https://mlsanity.com/paper/2012.13052/) (2020)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Learning Supervised Topic Models for Classification and Regression from Crowds](https://mlsanity.com/paper/1808.05902/) (2018)
- [A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control](https://mlsanity.com/paper/1812.02736/) (2018)

## Built on by (most cited first, 4 of 4)

- [Learning from Multiple Expert Annotators for Enhancing Anomaly Detection in Medical Image Analysis](https://mlsanity.com/paper/2203.10611/) (2022)
- [Learning from Noisy Crowd Labels with Logics](https://mlsanity.com/paper/2302.06337/) (2023)
- [Improving Object Detection in Medical Image Analysis through Multiple Expert Annotators: An Empirical Investigation](https://mlsanity.com/paper/2303.16507/) (2023)
- [Label Filling via Mixed Supervision for Medical Image Segmentation from Noisy Annotations](https://mlsanity.com/paper/2410.16057/) (2024)

## Abstract (arXiv)

Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of possibly noisy labels provided by multiple annotators under the influence of their varying abilities and own biases. In modeling the probability transition process from latent true labels to observed labels, most existing methods adopt class-level confusion matrices of annotators that observed labels do not depend on the instance features, just determined by the true labels. It may limit the performance that the classifier can achieve. In this work, we propose the noise transition matrix, which incorporates the influence of instance features on annotators' performance based on confusion matrices. Furthermore, we propose a simple yet effective learning framework, which consists of a classifier module and a noise transition matrix module in a unified neural network architecture. Experimental results demonstrate the superiority of our method in comparison with state-of-the-art methods.