# SeedBERT: Recovering Annotator Rating Distributions from an Aggregated Label

arXiv: https://arxiv.org/abs/2211.13196 · HTML: https://mlsanity.com/paper/2211.13196/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 23 Nov 2022.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Label Distribution Learning](https://mlsanity.com/paper/1408.6027/) (2014)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)

## Built on by (most cited first, 3 of 3)

- [Under the Surface: Tracking the Artifactuality of LLM-Generated Data](https://mlsanity.com/paper/2401.14698/) (2024)
- [Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions](https://mlsanity.com/paper/2404.11023/) (2024)
- [Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives](https://mlsanity.com/paper/2508.02853/) (2025)

## Abstract (arXiv)

Many machine learning tasks -- particularly those in affective computing -- are inherently subjective. When asked to classify facial expressions or to rate an individual's attractiveness, humans may disagree with one another, and no single answer may be objectively correct. However, machine learning datasets commonly have just one "ground truth" label for each sample, so models trained on these labels may not perform well on tasks that are subjective in nature. Though allowing models to learn from the individual annotators' ratings may help, most datasets do not provide annotator-specific labels for each sample. To address this issue, we propose SeedBERT, a method for recovering annotator rating distributions from a single label by inducing pre-trained models to attend to different portions of the input. Our human evaluations indicate that SeedBERT's attention mechanism is consistent with human sources of annotator disagreement. Moreover, in our empirical evaluations using large language models, SeedBERT demonstrates substantial gains in performance on downstream subjective tasks compared both to standard deep learning models and to other current models that account explicitly for annotator disagreement.