# Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions

arXiv: https://arxiv.org/abs/2503.22711 · HTML: https://mlsanity.com/paper/2503.22711/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models](https://mlsanity.com/paper/2306.05350/) (2023)
- [EMO-SUPERB: An In-depth Look at Speech Emotion Recognition](https://mlsanity.com/paper/2402.13018/) (2024)
- [Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotions](https://mlsanity.com/paper/2002.01323/) (2020)
- [Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition](https://mlsanity.com/paper/2201.11826/) (2022)
- [Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice](https://mlsanity.com/paper/1907.00112/) (2019)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation](https://mlsanity.com/paper/2207.03334/) (2022)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [Articulatory information and Multiview Features for Large Vocabulary Continuous Speech Recognition](https://mlsanity.com/paper/1802.05853/) (2018)
- [Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions](https://mlsanity.com/paper/2207.12135/) (2022)
- [Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech](https://mlsanity.com/paper/2407.13035/) (2024)
- [Investigation and Analysis of Hyper and Hypo neuron pruning to selectively update neurons during Unsupervised Adaptation](https://mlsanity.com/paper/2001.01755/) (2020)
- [The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability](https://mlsanity.com/paper/2408.11956/) (2024)

## Built on by (most cited first, 2 of 2)

- [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025)
- [Learning from Annotation Uncertainty: Entropy-Aware Curriculum for Speech Emotion Recognition](https://mlsanity.com/paper/2606.27536/) (2026)

## Abstract (arXiv)

Spontaneous speech emotion data usually contain perceptual grades where graders assign emotion score after listening to the speech files. Such perceptual grades introduce uncertainty in labels due to grader opinion variation. Grader variation is addressed by using consensus grades as groundtruth, where the emotion with the highest vote is selected. Consensus grades fail to consider ambiguous instances where a speech sample may contain multiple emotions, as captured through grader opinion uncertainty. We demonstrate that using the probability density function of the emotion grades as targets instead of the commonly used consensus grades, provide better performance on benchmark evaluation sets compared to results reported in the literature. We show that a saliency driven foundation model (FM) representation selection helps to train a state-of-the-art speech emotion model for both dimensional and categorical emotion recognition. Comparing representations obtained from different FMs, we observed that focusing on overall test-set performance can be deceiving, as it fails to reveal the models generalization capacity across speakers and gender. We demonstrate that performance evaluation across multiple test-sets and performance analysis across gender and speakers are useful in assessing usefulness of emotion models. Finally, we demonstrate that label uncertainty and data-skew pose a challenge to model evaluation, where instead of using the best hypothesis, it is useful to consider the 2- or 3-best hypotheses.