# Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions

arXiv: https://arxiv.org/abs/2207.12135 · HTML: https://mlsanity.com/paper/2207.12135/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 47 references that are in the map.
- First posted on arXiv 25 Jul 2022.

## Builds on (most cited first, 10 of 10)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Speech Emotion Recognition using Semantic Information](https://mlsanity.com/paper/2103.02993/) (2021)
- [End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks](https://mlsanity.com/paper/2110.03299/) (2021)
- [Defining and Quantifying Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2009.12842/) (2020)
- [Objective priors for the number of degrees of freedom of a multivariate t distribution and the t-copula](https://mlsanity.com/paper/1701.05638/) (2017)

## Built on by (most cited first, 4 of 4)

- [End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning](https://mlsanity.com/paper/2209.15449/) (2022)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions](https://mlsanity.com/paper/2503.22711/) (2025)
- [Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition](https://mlsanity.com/paper/2609.05755/) (2026)

## Abstract (arXiv)

As different people perceive others' emotional expressions differently, their annotation in terms of arousal and valence are per se subjective. To address this, these emotion annotations are typically collected by multiple annotators and averaged across annotators in order to obtain labels for arousal and valence. However, besides the average, also the uncertainty of a label is of interest, and should also be modeled and predicted for automatic emotion recognition. In the literature, for simplicity, label uncertainty modeling is commonly approached with a Gaussian assumption on the collected annotations. However, as the number of annotators is typically rather small due to resource constraints, we argue that the Gaussian approach is a rather crude assumption. In contrast, in this work we propose to model the label distribution using a Student's t-distribution which allows us to account for the number of annotations available. With this model, we derive the corresponding Kullback-Leibler divergence based loss function and use it to train an estimator for the distribution of emotion labels, from which the mean and uncertainty can be inferred. Through qualitative and quantitative analysis, we show the benefits of the t-distribution over a Gaussian distribution. We validate our proposed method on the AVEC'16 dataset. Results reveal that our t-distribution based approach improves over the Gaussian approach with state-of-the-art uncertainty modeling results in speech-based emotion recognition, along with an optimal and even faster convergence.