# Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis

arXiv: https://arxiv.org/abs/2312.16180 · HTML: https://mlsanity.com/paper/2312.16180/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition](https://mlsanity.com/paper/2008.06682/) (2020)
- [Detecting Emotion Primitives from Speech and their use in discerning Categorical Emotions](https://mlsanity.com/paper/2002.01323/) (2020)
- [Sentiment-Aware Automatic Speech Recognition pre-training for enhanced Speech Emotion Recognition](https://mlsanity.com/paper/2201.11826/) (2022)
- [Leveraging Acoustic Cues and Paralinguistic Embeddings to Detect Expression from Voice](https://mlsanity.com/paper/1907.00112/) (2019)
- [Speech Emotion: Investigating Model Representations, Multi-Task Learning and Knowledge Distillation](https://mlsanity.com/paper/2207.03334/) (2022)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions](https://mlsanity.com/paper/2207.12135/) (2022)
- [Investigation and Analysis of Hyper and Hypo neuron pruning to selectively update neurons during Unsupervised Adaptation](https://mlsanity.com/paper/2001.01755/) (2020)

## Built on by (most cited first, 7 of 7)

- [Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech](https://mlsanity.com/paper/2407.13035/) (2024)
- [Model-driven Heart Rate Estimation and Heart Murmur Detection based on Phonocardiogram](https://mlsanity.com/paper/2407.18424/) (2024)
- [Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions](https://mlsanity.com/paper/2503.22711/) (2025)
- [Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation](https://mlsanity.com/paper/2505.20745/) (2025)
- [Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affect](https://mlsanity.com/paper/2505.21809/) (2025)
- [Affect Models Have Weak Generalizability to Atypical Speech](https://mlsanity.com/paper/2504.16283/) (2025)
- [Breathing and Semantic Pause Detection and Exertion-Level Classification in Post-Exercise Speech](https://mlsanity.com/paper/2509.15473/) (2025)

## Abstract (arXiv)

Representations derived from models such as BERT (Bidirectional Encoder Representations from Transformers) and HuBERT (Hidden units BERT), have helped to achieve state-of-the-art performance in dimensional speech emotion recognition. Despite their large dimensionality, and even though these representations are not tailored for emotion recognition tasks, they are frequently used to train large speech emotion models with high memory and computational costs. In this work, we show that there exist lower-dimensional subspaces within the these pre-trained representational spaces that offer a reduction in downstream model complexity without sacrificing performance on emotion estimation. In addition, we model label uncertainty in the form of grader opinion variance, and demonstrate that such information can improve the models generalization capacity and robustness. Finally, we compare the robustness of the emotion models against acoustic degradations and observed that the reduced dimensional representations were able to retain the performance similar to the full-dimensional representations without significant regression in dimensional emotion performance.