# Deep neural networks for emotion recognition combining audio and transcripts

arXiv: https://arxiv.org/abs/1911.00432 · HTML: https://mlsanity.com/paper/1911.00432/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 11 of 11)

- [The VoicePrivacy 2024 Challenge Evaluation Plan](https://mlsanity.com/paper/2404.02677/) (2024)
- [Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition](https://mlsanity.com/paper/2201.06309/) (2022)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Leveraging Pre-trained Language Model for Speech Sentiment Analysis](https://mlsanity.com/paper/2106.06598/) (2021)
- [On the Impact of Word Error Rate on Acoustic-Linguistic Speech Emotion Recognition: An Update for the Deep Learning Era](https://mlsanity.com/paper/2104.10121/) (2021)
- [Beyond Isolated Utterances: Conversational Emotion Recognition](https://mlsanity.com/paper/2109.06112/) (2021)
- [Sentiment recognition of Italian elderly through domain adaptation on cross-corpus speech dataset](https://mlsanity.com/paper/2211.07307/) (2022)
- [Evaluation of Speaker Anonymization on Emotional Speech](https://mlsanity.com/paper/2305.01759/) (2023)
- [Position-based Contributive Embeddings for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2108.05098/) (2021)
- [Dialog speech sentiment classification for imbalanced datasets](https://mlsanity.com/paper/2109.07228/) (2021)
- [SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training](https://mlsanity.com/paper/2609.02941/) (2026)

## Abstract (arXiv)

In this paper, we propose to improve emotion recognition by combining acoustic information and conversation transcripts. On the one hand, an LSTM network was used to detect emotion from acoustic features like f0, shimmer, jitter, MFCC, etc. On the other hand, a multi-resolution CNN was used to detect emotion from word sequences. This CNN consists of several parallel convolutions with different kernel sizes to exploit contextual information at different levels. A temporal pooling layer aggregates the hidden representations of different words into a unique sequence level embedding, from which we computed the emotion posteriors. We optimized a weighted sum of classification and verification losses. The verification loss tries to bring embeddings from the same emotions closer while separating embeddings from different emotions. We also compared our CNN with state-of-the-art text-based hand-crafted features (e-vector). We evaluated our approach on the USC-IEMOCAP dataset as well as the dataset consisting of US English telephone speech. In the former, we used human-annotated transcripts while in the latter, we used ASR transcripts. The results showed fusing audio and transcript information improved unweighted accuracy by relative 24% for IEMOCAP and relative 3.4% for the telephone data compared to a single acoustic system.