# Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective

arXiv: https://arxiv.org/abs/2407.06902 · HTML: https://mlsanity.com/paper/2407.06902/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 78 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 30 of 33)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Tensor Decomposition for Signal Processing and Machine Learning](https://mlsanity.com/paper/1607.01668/) (2016)
- [Provably Robust DPO: Aligning Language Models with Noisy Feedback](https://mlsanity.com/paper/2403.00409/) (2024)
- [Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion](https://mlsanity.com/paper/1902.03680/) (2019)
- [MaxMin-RLHF: Alignment with Diverse Human Preferences](https://mlsanity.com/paper/2402.08925/) (2024)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Disentangling Human Error from the Ground Truth in Segmentation of Medical Images](https://mlsanity.com/paper/2007.15963/) (2020)
- [Nonnegative Matrix Factorization for Signal and Data Analytics: Identifiability, Algorithms, and Applications](https://mlsanity.com/paper/1803.01257/) (2018)
- [Aligning Crowd Feedback via Distributional Preference Reward Modeling](https://mlsanity.com/paper/2402.09764/) (2024)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Crowdsourcing via Pairwise Co-occurrences: Identifiability and Algorithms](https://mlsanity.com/paper/1909.12325/) (2019)
- [Adversarial Crowdsourcing Through Robust Rank-One Matrix Completion](https://mlsanity.com/paper/2010.12181/) (2020)
- [Unsupervised Ensemble Learning with Dependent Classifiers](https://mlsanity.com/paper/1510.05830/) (2015)
- [Estimating the Accuracies of Multiple Classifiers Without Labeled Data](https://mlsanity.com/paper/1407.7644/) (2014)
- [Tensors, Learning, and 'Kolmogorov Extension' for Finite-alphabet Random Vectors](https://mlsanity.com/paper/1712.00205/) (2017)
- [Deep Learning From Crowdsourced Labels: Coupled Cross-entropy Minimization, Identifiability, and Regularization](https://mlsanity.com/paper/2306.03288/) (2023)
- [Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing](https://mlsanity.com/paper/1408.1387/) (2014)
- [Exact Exponent in Optimal Rates for Crowdsourcing](https://mlsanity.com/paper/1605.07696/) (2016)
- [Recovering Joint Probability of Discrete Random Variables from Pairwise Marginals](https://mlsanity.com/paper/2006.16912/) (2020)
- [Crowdsourcing with Fairness, Diversity and Budget Constraints](https://mlsanity.com/paper/1810.13314/) (2018)
- [Blind Multiclass Ensemble Classification](https://mlsanity.com/paper/1712.02903/) (2017)
- [Crowd-PrefRL: Preference-Based Reward Learning from Crowds](https://mlsanity.com/paper/2401.10941/) (2024)
- [Crowdsourcing via Annotator Co-occurrence Imputation and Provable Symmetric Nonnegative Matrix Factorization](https://mlsanity.com/paper/2106.07193/) (2021)
- [Deep Clustering with Incomplete Noisy Pairwise Annotations: A Geometric Regularization Approach](https://mlsanity.com/paper/2305.19391/) (2023)
- [Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysis](https://mlsanity.com/paper/2304.12573/) (2023)
- [Reliable Crowdsourcing for Multi-Class Labeling using Coding Theory](https://mlsanity.com/paper/1309.3330/) (2013)

## Built on by (most cited first, 3 of 3)

- [Optimal Fair Aggregation of Crowdsourced Noisy Labels using Demographic Parity Constraints](https://mlsanity.com/paper/2601.23221/) (2026)
- [ASAT: Adaptive Scoring and Thresholding with Human Feedback for Robust Out-of-Distribution Detection](https://mlsanity.com/paper/2505.02299/) (2025)
- [On the Calibration of Crowdsourcing Aggregators](https://mlsanity.com/paper/2610.16815/) (2026)

## Abstract (arXiv)

One of the primary catalysts fueling advances in artificial intelligence (AI) and machine learning (ML) is the availability of massive, curated datasets. A commonly used technique to curate such massive datasets is crowdsourcing, where data are dispatched to multiple annotators. The annotator-produced labels are then fused to serve downstream learning and inference tasks. This annotation process often creates noisy labels due to various reasons, such as the limited expertise, or unreliability of annotators, among others. Therefore, a core objective in crowdsourcing is to develop methods that effectively mitigate the negative impact of such label noise on learning tasks. This feature article introduces advances in learning from noisy crowdsourced labels. The focus is on key crowdsourcing models and their methodological treatments, from classical statistical models to recent deep learning-based approaches, emphasizing analytical insights and algorithmic developments. In particular, this article reviews the connections between signal processing (SP) theory and methods, such as identifiability of tensor and nonnegative matrix factorization, and novel, principled solutions of longstanding challenges in crowdsourcing -- showing how SP perspectives drive the advancements of this field. Furthermore, this article touches upon emerging topics that are critical for developing cutting-edge AI/ML systems, such as crowdsourcing in reinforcement learning with human feedback (RLHF) and direct preference optimization (DPO) that are key techniques for fine-tuning large language models (LLMs).