# Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network

arXiv: https://arxiv.org/abs/1911.09075 · HTML: https://mlsanity.com/paper/1911.09075/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 30 of 38)

- [DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversations](https://mlsanity.com/paper/2106.01978/) (2021)
- [EmoBERTa: Speaker-Aware Emotion Recognition in Conversation with RoBERTa](https://mlsanity.com/paper/2108.12009/) (2021)
- [Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection](https://mlsanity.com/paper/2106.01071/) (2021)
- [A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.20494/) (2023)
- [CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation](https://mlsanity.com/paper/2108.11626/) (2021)
- [Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion](https://mlsanity.com/paper/2404.17858/) (2024)
- [Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations](https://mlsanity.com/paper/2407.00119/) (2024)
- [Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition](https://mlsanity.com/paper/2201.05834/) (2022)
- [Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations](https://mlsanity.com/paper/2306.01505/) (2023)
- [Discovering Emotion and Reasoning its Flip in Multi-Party Conversations using Masked Memory Network and Transformer](https://mlsanity.com/paper/2103.12360/) (2021)
- [CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition](https://mlsanity.com/paper/2307.15432/) (2023)
- [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023)
- [Deep Emotion Recognition in Textual Conversations: A Survey](https://mlsanity.com/paper/2211.09172/) (2022)
- [Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects](https://mlsanity.com/paper/2505.20511/) (2025)
- [Privacy-Preserving Video Classification with Convolutional Neural Networks](https://mlsanity.com/paper/2102.03513/) (2021)
- [Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognition](https://mlsanity.com/paper/2108.00194/) (2021)
- [DARER: Dual-task Temporal Relational Recurrent Reasoning Network for Joint Dialog Sentiment Classification and Act Recognition](https://mlsanity.com/paper/2203.03856/) (2022)
- [Brain-inspired Artificial Intelligence: A Comprehensive Review](https://mlsanity.com/paper/2408.14811/) (2024)
- [Relational Temporal Graph Reasoning for Dual-task Dialogue Language Understanding](https://mlsanity.com/paper/2306.09114/) (2023)
- [How you feelin'? Learning Emotions and Mental States in Movie Scenes](https://mlsanity.com/paper/2304.05634/) (2023)
- [Emotion Dynamics Modeling via BERT](https://mlsanity.com/paper/2104.07252/) (2021)
- [RBA-GCN: Relational Bilevel Aggregation Graph Convolutional Network for Emotion Recognition](https://mlsanity.com/paper/2308.11029/) (2023)
- [SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)](https://mlsanity.com/paper/2402.18944/) (2024)
- [Emotion Flip Reasoning in Multiparty Conversations](https://mlsanity.com/paper/2306.13959/) (2023)
- [Modelling Emotion Dynamics in Song Lyrics with State Space Models](https://mlsanity.com/paper/2210.09434/) (2022)
- [PAGE: A Position-Aware Graph-Based Model for Emotion Cause Entailment in Conversation](https://mlsanity.com/paper/2303.01795/) (2023)
- [From Multilingual Complexity to Emotional Clarity: Leveraging Commonsense to Unveil Emotions in Code-Mixed Dialogues](https://mlsanity.com/paper/2310.13080/) (2023)
- [AdCOFE: Advanced Contextual Feature Extraction in Conversations for emotion classification](https://mlsanity.com/paper/2104.04517/) (2021)
- [A Dual-Stream Recurrence-Attention Network With Global-Local Awareness for Emotion Recognition in Textual Dialog](https://mlsanity.com/paper/2307.00449/) (2023)
- [FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition](https://mlsanity.com/paper/2307.12221/) (2023)

## Abstract (arXiv)

Real-time emotion recognition (RTER) in conversations is significant for developing emotionally intelligent chatting machines. Without the future context in RTER, it becomes critical to build the memory bank carefully for capturing historical context and summarize the memories appropriately to retrieve relevant information. We propose an Attention Gated Hierarchical Memory Network (AGHMN) to address the problems of prior work: (1) Commonly used convolutional neural networks (CNNs) for utterance feature extraction are less compatible in the memory modules; (2) Unidirectional gated recurrent units (GRUs) only allow each historical utterance to have context before it, preventing information propagation in the opposite direction; (3) The Soft Attention for summarizing loses the positional and ordering information of memories, regardless of how the memory bank is built. Particularly, we propose a Hierarchical Memory Network (HMN) with a bidirectional GRU (BiGRU) as the utterance reader and a BiGRU fusion layer for the interaction between historical utterances. For memory summarizing, we propose an Attention GRU (AGRU) where we utilize the attention weights to update the internal state of GRU. We further promote the AGRU to a bidirectional variant (BiAGRU) to balance the contextual information from recent memories and that from distant memories. We conduct experiments on two emotion conversation datasets with extensive analysis, demonstrating the efficacy of our AGHMN models.