# Effect of Attention and Self-Supervised Speech Embeddings on Non-Semantic Speech Tasks

arXiv: https://arxiv.org/abs/2308.14359 · HTML: https://mlsanity.com/paper/2308.14359/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing](https://mlsanity.com/paper/2211.01756/) (2022)
- [The ACM Multimedia 2023 Computational Paralinguistics Challenge: Emotion Share & Requests](https://mlsanity.com/paper/2304.14882/) (2023)

## Built on by (most cited first, 4 of 4)

- [Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMs](https://mlsanity.com/paper/2506.00304/) (2025)
- [Phase-driven Domain Generalizable Learning for Nonstationary Time Series](https://mlsanity.com/paper/2402.05960/) (2024)
- [Missingness-resilient Video-enhanced Multimodal Disfluency Detection](https://mlsanity.com/paper/2406.06964/) (2024)
- [TimeSliver : Symbolic-Linear Decomposition for Explainable Time Series Classification](https://mlsanity.com/paper/2601.21289/) (2026)

## Abstract (arXiv)

Human emotion understanding is pivotal in making conversational technology mainstream. We view speech emotion understanding as a perception task which is a more realistic setting. With varying contexts (languages, demographics, etc.) different share of people perceive the same speech segment as a non-unanimous emotion. As part of the ACM Multimedia 2023 Computational Paralinguistics ChallengE (ComParE) in the EMotion Share track, we leverage their rich dataset of multilingual speakers and multi-label regression target of 'emotion share' or perception of that emotion. We demonstrate that the training scheme of different foundation models dictates their effectiveness for tasks beyond speech recognition, especially for non-semantic speech tasks like emotion understanding. This is a very complex task due to multilingual speakers, variability in the target labels, and inherent imbalance in the regression dataset. Our results show that HuBERT-Large with a self-attention-based light-weight sequence model provides 4.6% improvement over the reported baseline.