# Improving Label Quality by Jointly Modeling Items and Annotators

arXiv: https://arxiv.org/abs/2106.10600 · HTML: https://mlsanity.com/paper/2106.10600/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 20 Jun 2021.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Neighborhood-based Pooling for Population-level Label Distribution Learning](https://mlsanity.com/paper/2003.07406/) (2020)

## Built on by (most cited first, 3 of 3)

- [Differential Parity: Relative Fairness Between Two Sets of Decisions](https://mlsanity.com/paper/2112.11279/) (2021)
- [Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks](https://mlsanity.com/paper/2605.09955/) (2026)
- [Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment](https://mlsanity.com/paper/2608.30842/) (2026)

## Abstract (arXiv)

We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model. Earlier research along these lines has neither fully incorporated label distributions nor explored clustering by annotators only or data only. Our framework incorporates all of these properties as: (1) a graphical model designed to provide better ground truth estimates of annotator responses as input to any black box supervised learning algorithm, and (2) a standalone neural model whose internal structure captures many of the properties of the graphical model. We conduct supervised learning experiments using both models and compare them to the performance of one baseline and a state-of-the-art model.