# Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations

arXiv: https://arxiv.org/abs/2010.02495 · HTML: https://mlsanity.com/paper/2010.02495/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems](https://mlsanity.com/paper/2202.02912/) (2022)
- [Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models](https://mlsanity.com/paper/2403.12388/) (2024)
- [TD-EVAL: Revisiting Task-Oriented Dialogue Evaluation by Combining Turn-Level Precision with Dialogue-Level Comparisons](https://mlsanity.com/paper/2504.19982/) (2025)
- [Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues](https://mlsanity.com/paper/2305.16798/) (2023)
- [Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process](https://mlsanity.com/paper/2305.12594/) (2023)
- [A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis](https://mlsanity.com/paper/2109.08412/) (2021)
- [When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories](https://mlsanity.com/paper/2606.05414/) (2026)
- [Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs](https://mlsanity.com/paper/2306.03984/) (2023)
- [CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2403.19056/) (2024)
- [A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS](https://mlsanity.com/paper/2212.03817/) (2022)
- [Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs](https://mlsanity.com/paper/2404.12994/) (2024)
- [A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment Analysis](https://mlsanity.com/paper/2410.09556/) (2024)
- [Actionable Conversational Quality Indicators for Improving Task-Oriented Dialog Systems](https://mlsanity.com/paper/2109.11064/) (2021)
- ["Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems](https://mlsanity.com/paper/2411.17437/) (2024)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)
- [Reward-Driven Interaction: Enhancing Proactive Dialogue Agents through User Satisfaction Prediction](https://mlsanity.com/paper/2505.18731/) (2025)
- [Personalized Turn-Level User Conversation Satisfaction Benchmark](https://mlsanity.com/paper/2605.29711/) (2026)
- [Cognivia: A Cognitive Behavioral Therapy Copilot for Evidence-Based Mental Healthcare](https://mlsanity.com/paper/2607.25681/) (2026)
- [MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation](https://mlsanity.com/paper/2608.24189/) (2026)

## Abstract (arXiv)

Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce the generalizability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model automatically weighs each turn's contribution towards the estimated dialogue-level rating, implicitly encodes temporal dependencies, and removes the need to hand-craft features. On dialogues sampled from 28 Alexa domains, two dialogue systems and three user groups, the joint dialogue-level satisfaction estimation model achieved up to an absolute 27% (0.43->0.70) and 7% (0.63->0.70) improvement in linear correlation performance over baseline deep neural net and benchmark Gradient boosting regression models, respectively.