# Multi-domain Conversation Quality Evaluation via User Satisfaction Estimation

arXiv: https://arxiv.org/abs/1911.08567 · HTML: https://mlsanity.com/paper/1911.08567/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models](https://mlsanity.com/paper/2403.12388/) (2024)
- [EmoUS: Simulating User Emotions in Task-Oriented Dialogues](https://mlsanity.com/paper/2306.01579/) (2023)
- [Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process](https://mlsanity.com/paper/2305.12594/) (2023)
- [A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis](https://mlsanity.com/paper/2109.08412/) (2021)
- [Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems](https://mlsanity.com/paper/2204.07135/) (2022)
- [Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs](https://mlsanity.com/paper/2306.03984/) (2023)
- [A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS](https://mlsanity.com/paper/2212.03817/) (2022)
- [Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs](https://mlsanity.com/paper/2404.12994/) (2024)
- [Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems](https://mlsanity.com/paper/2109.11064/) (2021)
- ["Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems](https://mlsanity.com/paper/2411.17437/) (2024)
- [LLM-guided Plan and Retrieval: A Strategic Alignment for Interpretable User Satisfaction Estimation in Dialogue](https://mlsanity.com/paper/2503.04675/) (2025)
- [DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessment](https://mlsanity.com/paper/2310.16319/) (2023)

## Abstract (arXiv)

An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and employ annotation schemes with limited generalizability to conversations spanning multiple domains. To address these gaps, we created a new Response Quality annotation scheme, introduced five new domain-independent feature sets and experimented with six machine learning models to estimate User Satisfaction at both turn and dialogue level. Response Quality ratings achieved significantly high correlation (0.76) with explicit turn-level user ratings. Using the new feature sets we introduced, Gradient Boosting Regression model achieved best (rating [1-5]) prediction performance on 26 seen (linear correlation ~0.79) and one new multi-turn domain (linear correlation 0.67). We observed a 16% relative improvement (68% -> 79%) in binary ("satisfactory/dissatisfactory") class prediction accuracy of a domain-independent dialogue-level satisfaction estimation model after including predicted turn-level satisfaction ratings as features.