# Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspective

arXiv: https://arxiv.org/abs/2206.09315 · HTML: https://mlsanity.com/paper/2206.09315/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 116 references that are in the map.
- First posted on arXiv 19 Jun 2022.

## Builds on (most cited first, 12 of 12)

- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Data Programming: Creating Large Training Sets, Quickly](https://mlsanity.com/paper/1605.07723/) (2016)
- [Deep learning from crowds](https://mlsanity.com/paper/1709.01779/) (2017)
- [Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing](https://mlsanity.com/paper/1406.3824/) (2014)
- [Crowdsourcing in Computer Vision](https://mlsanity.com/paper/1611.02145/) (2016)
- [PU Learning for Matrix Completion](https://mlsanity.com/paper/1411.6081/) (2014)
- [Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds](https://mlsanity.com/paper/1905.13436/) (2019)
- [Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs](https://mlsanity.com/paper/2004.13465/) (2020)
- [Graph convolutional networks for learning with few clean and many noisy labels](https://mlsanity.com/paper/1910.00324/) (2019)
- [Active Multi-Label Crowd Consensus](https://mlsanity.com/paper/1911.02789/) (2019)
- [Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons](https://mlsanity.com/paper/1504.00064/) (2015)
- [Crowdsourcing for Bioinformatics](https://mlsanity.com/paper/1302.6667/) (2013)

## Built on by (most cited first, 11 of 11)

- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds](https://mlsanity.com/paper/2306.03116/) (2023)
- [Coupled Confusion Correction: Learning from Crowds with Sparse Annotations](https://mlsanity.com/paper/2312.07331/) (2023)
- [Multi-annotator Deep Learning: A Probabilistic Framework for Classification](https://mlsanity.com/paper/2304.02539/) (2023)
- [Learning with Confidence: Training Better Classifiers from Soft Labels](https://mlsanity.com/paper/2409.16071/) (2024)
- [SkillAggregation: Reference-free LLM-Dependent Aggregation](https://mlsanity.com/paper/2410.10215/) (2024)
- [CrowdAgent: Multi-Agent Managed Multi-Source Annotation System](https://mlsanity.com/paper/2509.14030/) (2025)
- [Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler](https://mlsanity.com/paper/2309.05086/) (2023)
- [Expertise-based Weighting for Regression Models with Noisy Labels](https://mlsanity.com/paper/2305.07430/) (2023)
- [Granular feedback merits sophisticated aggregation](https://mlsanity.com/paper/2507.12041/) (2025)
- [Unsupervised Ensemble Learning Through Deep Energy-based Models](https://mlsanity.com/paper/2601.20556/) (2026)

## Abstract (arXiv)

Big data have the characteristics of enormous volume, high velocity, diversity, value-sparsity, and uncertainty, which lead the knowledge learning from them full of challenges. With the emergence of crowdsourcing, versatile information can be obtained on-demand so that the wisdom of crowds is easily involved to facilitate the knowledge learning process. During the past thirteen years, researchers in the AI community made great efforts to remove the obstacles in the field of learning from crowds. This concentrated survey paper comprehensively reviews the technical progress in crowdsourcing learning from a systematic perspective that includes three dimensions of data, models, and learning processes. In addition to reviewing existing important work, the paper places a particular emphasis on providing some promising blueprints on each dimension as well as discussing the lessons learned from our past research work, which will light up the way for new researchers and encourage them to pursue new contributions.