# Emotional Conversation Generation with Heterogeneous Graph Neural Network

arXiv: https://arxiv.org/abs/2012.04882 · HTML: https://mlsanity.com/paper/2012.04882/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2212.07672/) (2022)
- [Deeper Insights into Deep Graph Convolutional Networks: Stability and Generalization](https://mlsanity.com/paper/2410.08473/) (2024)
- [D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization](https://mlsanity.com/paper/2305.12767/) (2023)
- [BJTU-WeChat's Systems for the WMT22 Chat Translation Task](https://mlsanity.com/paper/2211.15009/) (2022)
- [Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model](https://mlsanity.com/paper/2306.10241/) (2023)
- [Conversation Derailment Forecasting with Graph Convolutional Networks](https://mlsanity.com/paper/2306.12982/) (2023)
- [Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasets](https://mlsanity.com/paper/2308.07871/) (2023)
- [Conversational Human Audio-visual Talking Dialogue Generation](https://mlsanity.com/paper/2607.02799/) (2026)

## Abstract (arXiv)

The successful emotional conversation system depends on sufficient perception and appropriate expression of emotions. In a real-life conversation, humans firstly instinctively perceive emotions from multi-source information, including the emotion flow hidden in dialogue history, facial expressions, audio, and personalities of speakers. Then, they convey suitable emotions according to their personalities, but these multiple types of information are insufficiently exploited in emotional conversation fields. To address this issue, in this paper, we propose a heterogeneous graph-based model for emotional conversation generation. Firstly, we design a Heterogeneous Graph-Based Encoder to represent the conversation content (i.e., the dialogue history, its emotion flow, facial expressions, audio, and speakers' personalities) with a heterogeneous graph neural network, and then predict suitable emotions for feedback. Secondly, we employ an Emotion-Personality-Aware Decoder to generate a response relevant to the conversation context as well as with appropriate emotions, through taking the encoded graph representations, the predicted emotions by the encoder and the personality of the current speaker as inputs. Experiments on both automatic and human evaluation show that our method can effectively perceive emotions from multi-source knowledge and generate a satisfactory response. Furthermore, based on the up-to-date text generator BART, our model still can achieve consistent improvement, which significantly outperforms some existing state-of-the-art models.