# Empathetic Dialogue Generation with Pre-trained RoBERTa-GPT2 and External Knowledge

arXiv: https://arxiv.org/abs/2109.03004 · HTML: https://mlsanity.com/paper/2109.03004/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1907.05774/) (2019)
- [Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations](https://mlsanity.com/paper/1909.10681/) (2019)
- [MoEL: Mixture of Empathetic Listeners](https://mlsanity.com/paper/1908.07687/) (2019)
- [Few-shot Natural Language Generation for Task-Oriented Dialog](https://mlsanity.com/paper/2002.12328/) (2020)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [MojiTalk: Generating Emotional Responses at Scale](https://mlsanity.com/paper/1711.04090/) (2017)
- [Affect-Driven Dialog Generation](https://mlsanity.com/paper/1904.02793/) (2019)
- [CDL: Curriculum Dual Learning for Emotion-Controllable Response Generation](https://mlsanity.com/paper/2005.00329/) (2020)
- [Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction](https://mlsanity.com/paper/2106.03044/) (2021)
- [EmpTransfo: A Multi-head Transformer Architecture for Creating Empathetic Dialog Systems](https://mlsanity.com/paper/2003.02958/) (2020)
- [Augmenting End-to-End Dialog Systems with Commonsense Knowledge](https://mlsanity.com/paper/1709.05453/) (2017)
- [Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset](https://mlsanity.com/paper/1811.00207/) (2018)

## Built on by (most cited first, 8 of 8)

- [Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities](https://mlsanity.com/paper/2206.05017/) (2022)
- [Knowledge-enhanced Memory Model for Emotional Support Conversation](https://mlsanity.com/paper/2310.07700/) (2023)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [CAB: Empathetic Dialogue Generation with Cognition, Affection and Behavior](https://mlsanity.com/paper/2302.01935/) (2023)
- [Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue History](https://mlsanity.com/paper/2206.08039/) (2022)
- [Acknowledgment of Emotional States: Generating Validating Responses for Empathetic Dialogue](https://mlsanity.com/paper/2402.12770/) (2024)
- [Unified Conversational Models with System-Initiated Transitions between Chit-Chat and Task-Oriented Dialogues](https://mlsanity.com/paper/2307.01664/) (2023)
- [ConceptNet infused DialoGPT for Underlying Commonsense Understanding and Reasoning in Dialogue Response Generation](https://mlsanity.com/paper/2209.15109/) (2022)

## Abstract (arXiv)

One challenge for dialogue agents is to recognize feelings of the conversation partner and respond accordingly. In this work, RoBERTa-GPT2 is proposed for empathetic dialogue generation, where the pre-trained auto-encoding RoBERTa is utilised as encoder and the pre-trained auto-regressive GPT-2 as decoder. With the combination of the pre-trained RoBERTa and GPT-2, our model realizes a new state-of-the-art emotion accuracy. To enable the empathetic ability of RoBERTa-GPT2 model, we propose a commonsense knowledge and emotional concepts extractor, in which the commonsensible and emotional concepts of dialogue context are extracted for the GPT-2 decoder. The experiment results demonstrate that the empathetic dialogue generation benefits from both pre-trained encoder-decoder architecture and external knowledge.