# Speech Emotion Recognition using Semantic Information

arXiv: https://arxiv.org/abs/2103.02993 · HTML: https://mlsanity.com/paper/2103.02993/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2021.

## Built on by (most cited first, 11 of 11)

- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition](https://mlsanity.com/paper/2307.10757/) (2023)
- [End-To-End Label Uncertainty Modeling for Speech-based Arousal Recognition Using Bayesian Neural Networks](https://mlsanity.com/paper/2110.03299/) (2021)
- [End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning](https://mlsanity.com/paper/2209.15449/) (2022)
- [Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models](https://mlsanity.com/paper/2305.19184/) (2023)
- [End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations](https://mlsanity.com/paper/2310.02281/) (2023)
- [Label Uncertainty Modeling and Prediction for Speech Emotion Recognition using t-Distributions](https://mlsanity.com/paper/2207.12135/) (2022)
- [STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition](https://mlsanity.com/paper/2402.01227/) (2024)
- [Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios](https://mlsanity.com/paper/2404.07484/) (2024)
- [Lightweight Models for Emotional Analysis in Video](https://mlsanity.com/paper/2503.10530/) (2025)
- [Unrequited Emotions: Investigating the Gaps in Motivation and Practice in Speech Emotion Recognition Research](https://mlsanity.com/paper/2604.25776/) (2026)

## Abstract (arXiv)

Speech emotion recognition is a crucial problem manifesting in a multitude of applications such as human computer interaction and education. Although several advancements have been made in the recent years, especially with the advent of Deep Neural Networks (DNN), most of the studies in the literature fail to consider the semantic information in the speech signal. In this paper, we propose a novel framework that can capture both the semantic and the paralinguistic information in the signal. In particular, our framework is comprised of a semantic feature extractor, that captures the semantic information, and a paralinguistic feature extractor, that captures the paralinguistic information. Both semantic and paraliguistic features are then combined to a unified representation using a novel attention mechanism. The unified feature vector is passed through a LSTM to capture the temporal dynamics in the signal, before the final prediction. To validate the effectiveness of our framework, we use the popular SEWA dataset of the AVEC challenge series and compare with the three winning papers. Our model provides state-of-the-art results in the valence and liking dimensions.