# Dialogue Quality and Emotion Annotations for Customer Support Conversations

arXiv: https://arxiv.org/abs/2311.13910 · HTML: https://mlsanity.com/paper/2311.13910/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 23 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems](https://mlsanity.com/paper/1704.00057/) (2017)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [Learning an Unreferenced Metric for Online Dialogue Evaluation](https://mlsanity.com/paper/2005.00583/) (2020)
- [Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2011.00483/) (2020)
- [TWEETSUMM -- A Dialog Summarization Dataset for Customer Service](https://mlsanity.com/paper/2111.11894/) (2021)
- [Automatic Evaluation and Moderation of Open-domain Dialogue Systems](https://mlsanity.com/paper/2111.02110/) (2021)

## Built on by (most cited first, 4 of 4)

- [Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models](https://mlsanity.com/paper/2404.14772/) (2024)
- [Infusing Emotions into Task-oriented Dialogue Systems: Understanding, Management, and Generation](https://mlsanity.com/paper/2408.02417/) (2024)
- [Findings of the WMT 2024 Shared Task on Chat Translation](https://mlsanity.com/paper/2410.11624/) (2024)
- [A Context-aware Framework for Translation-mediated Conversations](https://mlsanity.com/paper/2412.04205/) (2024)

## Abstract (arXiv)

Task-oriented conversational datasets often lack topic variability and linguistic diversity. However, with the advent of Large Language Models (LLMs) pretrained on extensive, multilingual and diverse text data, these limitations seem overcome. Nevertheless, their generalisability to different languages and domains in dialogue applications remains uncertain without benchmarking datasets. This paper presents a holistic annotation approach for emotion and conversational quality in the context of bilingual customer support conversations. By performing annotations that take into consideration the complete instances that compose a conversation, one can form a broader perspective of the dialogue as a whole. Furthermore, it provides a unique and valuable resource for the development of text classification models. To this end, we present benchmarks for Emotion Recognition and Dialogue Quality Estimation and show that further research is needed to leverage these models in a production setting.