# End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2110.03299 · HTML: https://mlsanity.com/paper/2110.03299/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 5 of 5)

- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [Speech Emotion Recognition using Semantic Information](https://mlsanity.com/paper/2103.02993/) (2021)
- [Defining and Quantifying Conversation Quality in Spontaneous Interactions](https://mlsanity.com/paper/2009.12842/) (2020)

## Built on by (most cited first, 9 of 9)

- [End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning](https://mlsanity.com/paper/2209.15449/) (2022)
- [EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data](https://mlsanity.com/paper/2309.07828/) (2023)
- [Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models](https://mlsanity.com/paper/2305.19184/) (2023)
- [Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions](https://mlsanity.com/paper/2207.12135/) (2022)
- [In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesis](https://mlsanity.com/paper/2306.01916/) (2023)
- [The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability](https://mlsanity.com/paper/2408.11956/) (2024)
- [Integrating Uncertainty into Neural Network-based Speech Enhancement](https://mlsanity.com/paper/2305.08744/) (2023)
- [Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction](https://mlsanity.com/paper/2604.07198/) (2026)
- [TargetSEC: Plug-and-Play In-the-Wild Speech Emotion Conversion via Arousal-Conditioned Latent Style Diffusion](https://mlsanity.com/paper/2606.07293/) (2026)

## Abstract (arXiv)

Emotions are subjective constructs. Recent end-to-end speech emotion recognition systems are typically agnostic to the subjective nature of emotions, despite their state-of-the-art performance. In this work, we introduce an end-to-end Bayesian neural network architecture to capture the inherent subjectivity in the arousal dimension of emotional expressions. To the best of our knowledge, this work is the first to use Bayesian neural networks for speech emotion recognition. At training, the network learns a distribution of weights to capture the inherent uncertainty related to subjective arousal annotations. To this end, we introduce a loss term that enables the model to be explicitly trained on a distribution of annotations, rather than training them exclusively on mean or gold-standard labels. We evaluate the proposed approach on the AVEC'16 dataset. Qualitative and quantitative analysis of the results reveals that the proposed model can aptly capture the distribution of subjective arousal annotations, with state-of-the-art results in mean and standard deviation estimations for uncertainty modeling.