# Bimodal Speech Emotion Recognition Using Pre-Trained Language Models

arXiv: https://arxiv.org/abs/1912.02610 · HTML: https://mlsanity.com/paper/1912.02610/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2019.

## Built on by (most cited first, 5 of 5)

- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [Leveraging Label Information for Multimodal Emotion Recognition](https://mlsanity.com/paper/2309.02106/) (2023)
- [Capturing Spectral and Long-term Contextual Information for Speech Emotion Recognition Using Deep Learning Techniques](https://mlsanity.com/paper/2308.04517/) (2023)

## Abstract (arXiv)

Speech emotion recognition is a challenging task and an important step towards more natural human-machine interaction. We show that pre-trained language models can be fine-tuned for text emotion recognition, achieving an accuracy of 69.5% on Task 4A of SemEval 2017, improving upon the previous state of the art by over 3% absolute. We combine these language models with speech emotion recognition, achieving results of 73.5% accuracy when using provided transcriptions and speech data on a subset of four classes of the IEMOCAP dataset. The use of noise-induced transcriptions and speech data results in an accuracy of 71.4%. For our experiments, we created IEmoNet, a modular and adaptable bimodal framework for speech emotion recognition based on pre-trained language models. Lastly, we discuss the idea of using an emotional classifier as a reward for reinforcement learning as a step towards more successful and convenient human-machine interaction.