# The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability

arXiv: https://arxiv.org/abs/2408.11956 · HTML: https://mlsanity.com/paper/2408.11956/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 22 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Evaluation of Error and Correlation-Based Loss Functions For Multitask Learning Dimensional Speech Emotion Recognition](https://mlsanity.com/paper/2003.10724/) (2020)
- [Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression](https://mlsanity.com/paper/2306.06760/) (2023)
- [End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks](https://mlsanity.com/paper/2110.03299/) (2021)

## Built on by (most cited first, 3 of 3)

- [Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions](https://mlsanity.com/paper/2503.22711/) (2025)
- [More Similar than Dissimilar: Modeling Annotators for Cross-Corpus Speech Emotion Recognition](https://mlsanity.com/paper/2509.12295/) (2025)
- [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025)

## Abstract (arXiv)

Emotion expression and perception are nuanced, complex, and highly subjective processes. When multiple annotators label emotional data, the resulting labels contain high variability. Most speech emotion recognition tasks address this by averaging annotator labels as ground truth. However, this process omits the nuance of emotion and inter-annotator variability, which are important signals to capture. Previous work has attempted to learn distributions to capture emotion variability, but these methods also lose information about the individual annotators. We address these limitations by learning to predict individual annotators and by introducing a novel method to create distributions from continuous model outputs that permit the learning of emotion distributions during model training. We show that this combined approach can result in emotion distributions that are more accurate than those seen in prior work, in both within- and cross-corpus settings.