# Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing

arXiv: https://arxiv.org/abs/2211.01756 · HTML: https://mlsanity.com/paper/2211.01756/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2022.

## Built on by (most cited first, 10 of 10)

- [What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model](https://mlsanity.com/paper/2306.06524/) (2023)
- [Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Tasks](https://mlsanity.com/paper/2308.14359/) (2023)
- [Adapting WavLM for Speech Emotion Recognition](https://mlsanity.com/paper/2405.04485/) (2024)
- [Sounding Like a Winner? Prosodic Differences in Post-Match Interviews](https://mlsanity.com/paper/2506.02283/) (2025)
- [Frame-level emotional state alignment method for speech emotion recognition](https://mlsanity.com/paper/2312.16383/) (2023)
- [Investigating the Impact of Word Informativeness on Speech Emotion Recognition](https://mlsanity.com/paper/2506.02239/) (2025)
- [iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis](https://mlsanity.com/paper/2602.21464/) (2026)
- [CochCeps-Augment: A Novel Self-Supervised Contrastive Learning Using Cochlear Cepstrum-based Masking for Speech Emotion Recognition](https://mlsanity.com/paper/2402.06923/) (2024)
- [Beyond the Labels: Unveiling Text-Dependency in Paralinguistic Speech Recognition Datasets](https://mlsanity.com/paper/2403.07767/) (2024)
- [Geometric Second-Order Feature Correlation Learning for Self-Supervised Speech Emotion Recognition](https://mlsanity.com/paper/2606.06550/) (2026)

## Abstract (arXiv)

When recognizing emotions from speech, we encounter two common problems: how to optimally capture emotion-relevant information from the speech signal and how to best quantify or categorize the noisy subjective emotion labels. Self-supervised pre-trained representations can robustly capture information from speech enabling state-of-the-art results in many downstream tasks including emotion recognition. However, better ways of aggregating the information across time need to be considered as the relevant emotion information is likely to appear piecewise and not uniformly across the signal. For the labels, we need to take into account that there is a substantial degree of noise that comes from the subjective human annotations. In this paper, we propose a novel approach to attentive pooling based on correlations between the representations' coefficients combined with label smoothing, a method aiming to reduce the confidence of the classifier on the training labels. We evaluate our proposed approach on the benchmark dataset IEMOCAP, and demonstrate high performance surpassing that in the literature. The code to reproduce the results is available at github.com/skakouros/s3prl_attentive_correlation.