# Beyond Isolated Utterances: Conversational Emotion Recognition

arXiv: https://arxiv.org/abs/2109.06112 · HTML: https://mlsanity.com/paper/2109.06112/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [x-vectors meet emotions: A study on dependencies between emotion and speaker recognition](https://mlsanity.com/paper/2002.05039/) (2020)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models](https://mlsanity.com/paper/1911.09762/) (2019)
- [Deep neural networks for emotion recognition combining audio and transcripts](https://mlsanity.com/paper/1911.00432/) (2019)
- [CopyPaste: An Augmentation Method for Speech Emotion Recognition](https://mlsanity.com/paper/2010.14602/) (2020)

## Built on by (most cited first, 1 of 1)

- [Distribution-based Emotion Recognition in Conversation](https://mlsanity.com/paper/2211.04834/) (2022)

## Abstract (arXiv)

Speech emotion recognition is the task of recognizing the speaker's emotional state given a recording of their utterance. While most of the current approaches focus on inferring emotion from isolated utterances, we argue that this is not sufficient to achieve conversational emotion recognition (CER) which deals with recognizing emotions in conversations. In this work, we propose several approaches for CER by treating it as a sequence labeling task. We investigated transformer architecture for CER and, compared it with ResNet-34 and BiLSTM architectures in both contextual and context-less scenarios using IEMOCAP corpus. Based on the inner workings of the self-attention mechanism, we proposed DiverseCatAugment (DCA), an augmentation scheme, which improved the transformer model performance by an absolute 3.3% micro-f1 on conversations and 3.6% on isolated utterances. We further enhanced the performance by introducing an interlocutor-aware transformer model where we learn a dictionary of interlocutor index embeddings to exploit diarized conversations.