# Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts

arXiv: https://arxiv.org/abs/2112.01938 · HTML: https://mlsanity.com/paper/2112.01938/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [The emotional arcs of stories are dominated by six basic shapes](https://mlsanity.com/paper/1606.07772/) (2016)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [Affect-Driven Dialog Generation](https://mlsanity.com/paper/1904.02793/) (2019)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines](https://mlsanity.com/paper/1803.07427/) (2018)
- [Adapting a Language Model for Controlled Affective Text Generation](https://mlsanity.com/paper/2011.04000/) (2020)
- [An End-to-End Network for Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2103.01544/) (2021)
- [Deep learning for affective computing: text-based emotion recognition in decision support](https://mlsanity.com/paper/1803.06397/) (2018)
- [Fine-Grained Emotion Prediction by Modeling Emotion Definitions](https://mlsanity.com/paper/2107.12135/) (2021)
- [Domain Authoring Assistant for Intelligent Virtual Agents](https://mlsanity.com/paper/1904.03266/) (2019)
- [Improving speech recognition by revising gated recurrent units](https://mlsanity.com/paper/1710.00641/) (2017)

## Built on by (most cited first, 9 of 9)

- [COGMEN: COntextualized GNN based Multimodal Emotion recognitioN](https://mlsanity.com/paper/2205.02455/) (2022)
- [CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition](https://mlsanity.com/paper/2307.15432/) (2023)
- [Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition](https://mlsanity.com/paper/2308.04502/) (2023)
- [Multi-Task Learning Framework for Extracting Emotion Cause Span and Entailment in Conversations](https://mlsanity.com/paper/2211.03742/) (2022)
- [Dynamic Causal Disentanglement Model for Dialogue Emotion Detection](https://mlsanity.com/paper/2309.06928/) (2023)
- [AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations](https://mlsanity.com/paper/2401.15164/) (2024)
- [AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning](https://mlsanity.com/paper/2402.10921/) (2024)
- [IITK at SemEval-2024 Task 10: Who is the speaker? Improving Emotion Recognition and Flip Reasoning in Conversations via Speaker Embeddings](https://mlsanity.com/paper/2404.04525/) (2024)
- [Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry](https://mlsanity.com/paper/2609.09924/) (2026)

## Abstract (arXiv)

Emotion Recognition in Conversations (ERC) is an important and active research area. Recent work has shown the benefits of using multiple modalities (e.g., text, audio, and video) for the ERC task. In a conversation, participants tend to maintain a particular emotional state unless some stimuli evokes a change. There is a continuous ebb and flow of emotions in a conversation. Inspired by this observation, we propose a multimodal ERC model and augment it with an emotion-shift component that improves performance. The proposed emotion-shift component is modular and can be added to any existing multimodal ERC model (with a few modifications). We experiment with different variants of the model, and results show that the inclusion of emotion shift signal helps the model to outperform existing models for ERC on MOSEI and IEMOCAP datasets.