# Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge

arXiv: https://arxiv.org/abs/2406.06646 · HTML: https://mlsanity.com/paper/2406.06646/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2024.

## Built on by (most cited first, 1 of 1)

- [Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction](https://mlsanity.com/paper/2412.18748/) (2024)

## Abstract (arXiv)

Speech Self-Supervised Learning (SSL) has demonstrated considerable efficacy in various downstream tasks. Nevertheless, prevailing self-supervised models often overlook the incorporation of emotion-related prior information, thereby neglecting the potential enhancement of emotion task comprehension through emotion prior knowledge in speech. In this paper, we propose an emotion-aware speech representation learning with intensity knowledge. Specifically, we extract frame-level emotion intensities using an established speech-emotion understanding model. Subsequently, we propose a novel emotional masking strategy (EMS) to incorporate emotion intensities into the masking process. We selected two representative models based on Transformer and CNN, namely MockingJay and Non-autoregressive Predictive Coding (NPC), and conducted experiments on IEMOCAP dataset. Experiments have demonstrated that the representations derived from our proposed method outperform the original model in SER task.