mlsanityData to 7 Oct 2026

DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues

arXiv 2103.01287 ↗First posted 1 Mar 2021cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Digital assistants are experiencing rapid growth due to their ability to assist users with day-to-day tasks where most dialogues are happening multi-turn. However, evaluating multi-turn dialogues remains challenging, especially at scale. We suggest a context-sensitive method to estimate the turn-level satisfaction for dialogue considering various types of user preferences. The costs of interactions between users and dialogue systems are formulated using a budget consumption concept. We assume users have an initial interaction budget for a dialogue formed based on the task complexity and that each turn has a cost. When the task is completed, or the budget has been exhausted, users quit the dialogue. We demonstrate our method's effectiveness by extensive experimentation with a simulated dialogue platform and real multi-turn dialogues.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

20 of 42 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  4. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  5. 2020Recipes for building an open-domain chatbotcited by 593
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  8. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  9. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  10. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  11. 2016A User Simulator for Task-Completion Dialoguescited by 56
  12. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
8 more
  1. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialogcited by 37
  2. 2019ConvLab: Multi-Domain End-to-End Dialog System Platformcited by 29
  3. 2017Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Modelscited by 26
  4. 2019The Eighth Dialog System Technology Challengecited by 25
  5. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
  6. 2018Adversarial Learning of Task-Oriented Neural Dialog Modelscited by 17
  7. 2018Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learningcited by 11
  8. 2020Predicting User Engagement Status for Online Evaluation of Intelligent Assistantscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 0; 2024: 4; 2025: 0; 2026: 1.

7 papers.

  1. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  2. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 15
  3. 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applicationscited by 5
  4. 2022SelF-Eval: Self-supervised Fine-grained Dialogue Evaluationcited by 2
  5. 2024LLM-Human Pipeline for Cultural Context Grounding of Conversationscited by 2
  6. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1
  7. 2026Personalized Turn-Level User Conversation Satisfaction Benchmarkcited by 0