# Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems

arXiv: https://arxiv.org/abs/2109.11064 · HTML: https://mlsanity.com/paper/2109.11064/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [MoEL: Mixture of Empathetic Listeners](https://mlsanity.com/paper/1908.07687/) (2019)
- [Proactive Human-Machine Conversation with Explicit Conversation Goals](https://mlsanity.com/paper/1906.05572/) (2019)
- [Towards Exploiting Background Knowledge for Building Conversation Systems](https://mlsanity.com/paper/1809.08205/) (2018)
- [A Network Perspective on Stratification of Multi-Label Data](https://mlsanity.com/paper/1704.08756/) (2017)
- [Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols](https://mlsanity.com/paper/2006.06110/) (2020)
- [Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations](https://mlsanity.com/paper/2010.02495/) (2020)
- [An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation](https://mlsanity.com/paper/1808.08795/) (2018)
- [Improving Knowledge-aware Dialogue Generation via Knowledge Base Question Answering](https://mlsanity.com/paper/1912.07491/) (2019)
- [Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation](https://mlsanity.com/paper/1911.08567/) (2019)
- [A Syntactically Constrained Bidirectional-Asynchronous Approach for Emotional Conversation Generation](https://mlsanity.com/paper/1806.07000/) (2018)
- [Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learning](https://mlsanity.com/paper/2001.07615/) (2020)

## Built on by (most cited first, 1 of 1)

- [Signals: Trajectory Sampling and Triage for Agentic Interactions](https://mlsanity.com/paper/2604.00356/) (2026)

## Abstract (arXiv)

Automatic dialog systems have become a mainstream part of online customer service. Many such systems are built, maintained, and improved by customer service specialists, rather than dialog systems engineers and computer programmers. As conversations between people and machines become commonplace, it is critical to understand what is working, what is not, and what actions can be taken to reduce the frequency of inappropriate system responses. These analyses and recommendations need to be presented in terms that directly reflect the user experience rather than the internal dialog processing. This paper introduces and explains the use of Actionable Conversational Quality Indicators (ACQIs), which are used both to recognize parts of dialogs that can be improved, and to recommend how to improve them. This combines benefits of previous approaches, some of which have focused on producing dialog quality scoring while others have sought to categorize the types of errors the dialog system is making. We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog systems used in commercial customer service applications, and on the publicly available CMU LEGOv2 conversational dataset (Raux et al. 2005). We report on the annotation and analysis of conversational datasets showing which ACQIs are important to fix in various situations. The annotated datasets are then used to build a predictive model which uses a turn-based vector embedding of the message texts and achieves an 79% weighted average f1-measure at the task of finding the correct ACQI for a given conversation. We predict that if such a model worked perfectly, the range of potential improvement actions a bot-builder must consider at each turn could be reduced by an average of 81%.