# Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

arXiv: https://arxiv.org/abs/2204.11320 · HTML: https://mlsanity.com/paper/2204.11320/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 24 Apr 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [EmpTransfo: A Multi-head Transformer Architecture for Creating Empathetic Dialog Systems](https://mlsanity.com/paper/2003.02958/) (2020)
- [Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media](https://mlsanity.com/paper/1803.02952/) (2018)
- [Reinforcement Learning Based Emotional Editing Constraint Conversation Generation](https://mlsanity.com/paper/1904.08061/) (2019)
- [Delivering Cognitive Behavioral Therapy Using A Conversational SocialRobot](https://mlsanity.com/paper/1909.06670/) (2019)
- [Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation](https://mlsanity.com/paper/2101.04030/) (2021)

## Built on by (most cited first, 5 of 5)

- [Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling](https://mlsanity.com/paper/2312.11947/) (2023)
- [OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model](https://mlsanity.com/paper/2507.05177/) (2025)
- [SYNTHEMPATHY: A Scalable Empathy Corpus Generated Using LLMs Without Any Crowdsourcing](https://mlsanity.com/paper/2502.17857/) (2025)
- [Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health Conversation](https://mlsanity.com/paper/2604.05795/) (2026)
- [Adaptation of domain-specific transformer models with text oversampling for sentiment analysis of social media posts on Covid-19 vaccines](https://mlsanity.com/paper/2209.10966/) (2022)

## Abstract (arXiv)

Modern day conversational agents are trained to emulate the manner in which humans communicate. To emotionally bond with the user, these virtual agents need to be aware of the affective state of the user. Transformers are the recent state of the art in sequence-to-sequence learning that involves training an encoder-decoder model with word embeddings from utterance-response pairs. We propose an emotion-aware transformer encoder for capturing the emotional quotient in the user utterance in order to generate human-like empathetic responses. The contributions of our paper are as follows: 1) An emotion detector module trained on the input utterances determines the affective state of the user in the initial phase 2) A novel transformer encoder is proposed that adds and normalizes the word embedding with emotion embedding thereby integrating the semantic and affective aspects of the input utterance 3) The encoder and decoder stacks belong to the Transformer-XL architecture which is the recent state of the art in language modeling. Experimentation on the benchmark Facebook AI empathetic dialogue dataset confirms the efficacy of our model from the higher BLEU-4 scores achieved for the generated responses as compared to existing methods. Emotionally intelligent virtual agents are now a reality and inclusion of affect as a modality in all human-machine interfaces is foreseen in the immediate future.