# Domain-Independent turn-level Dialogue Quality Evaluation via User Satisfaction Estimation

arXiv: https://arxiv.org/abs/1908.07064 · HTML: https://mlsanity.com/paper/1908.07064/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2019.

## Built on by (most cited first, 5 of 5)

- [A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis](https://mlsanity.com/paper/2109.08412/) (2021)
- [High Quality Related Search Query Suggestions using Deep Reinforcement Learning](https://mlsanity.com/paper/2108.04452/) (2021)
- [Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs](https://mlsanity.com/paper/2306.03984/) (2023)
- ["Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems](https://mlsanity.com/paper/2411.17437/) (2024)
- [ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2601.11854/) (2026)

## Abstract (arXiv)

An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and rely on annotation schemes with low inter-rater reliability, limiting generalizability to conversations spanning multiple domains. To address these gaps, we created a new Response Quality annotation scheme, based on which we developed turn-level User Satisfaction metric. We introduced five new domain-independent feature sets and experimented with six machine learning models to estimate the new satisfaction metric. Using Response Quality annotation scheme, across randomly sampled single and multi-turn conversations from 26 domains, we achieved high inter-annotator agreement (Spearman's rho 0.94). The Response Quality labels were highly correlated (0.76) with explicit turn-level user ratings. Gradient boosting regression achieved best correlation of ~0.79 between predicted and annotated user satisfaction labels. Multi Layer Perceptron and Gradient Boosting regression models generalized to an unseen domain better (linear correlation 0.67) than other models. Finally, our ablation study verified that our novel features significantly improved model performance.