mlsanityData to 14 Oct 2026

What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation

arXiv 2203.13927 ↗First posted 25 Mar 2022cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, which is cumbersome to collect. In this work, we propose to use information that can be automatically extracted from the next user utterance, such as its sentiment or whether the user explicitly ends the conversation, as a proxy to measure the quality of the previous system response. This allows us to train on a massive set of dialogs with weak supervision, without requiring manual system turn quality annotations. Experiments show that our model is comparable to models trained on human annotated data. Furthermore, our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 25 Mar 2022.

Builds on

23 of 34 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  5. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  6. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  7. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  8. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  9. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  10. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  11. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  12. 2018Conversational AI: The Science Behind the Alexa Prizecited by 63
11 more
  1. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  2. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  3. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  4. 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
  5. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  6. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  7. 2020Emora: An Inquisitive Social Chatbot Who Cares For Youcited by 16
  8. 2020Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversationscited by 5
  9. 2019Beyond Turing: Intelligent Agents Centered on the Usercited by 4
  10. 2021Athena 2.0: Contextualized Dialogue Management for an Alexa Prize SocialBotcited by 4
  11. 2021What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impactscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 2; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  3. 2023Paralinguistics-Enhanced Large Language Modeling of Spoken Dialoguecited by 25
  4. 2024Self-seeding and Multi-intent Self-instructing LLMs for Generating Intent-aware Information-Seeking dialogscited by 18
  5. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  6. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  7. 2023ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systemscited by 11
  8. 2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
  9. 2023A Transformer-based Response Evaluator for Open-Domain Spoken Conversationcited by 4
  10. 2023Psychological Metrics for Dialog System Evaluationcited by 4
  11. 2022Open-Domain Dialog Evaluation using Follow-Ups Likelihoodcited by 3
  12. 2023Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogscited by 3
6 more
  1. 2023CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogscited by 3
  2. 2025Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12cited by 1
  3. 2023Embedding Mental Health Discourse for Community Recommendationcited by 0
  4. 2024Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systemscited by 0
  5. 2025Prompt reinforcing for long-term planning of large language modelscited by 0
  6. 2026Measuring Semantic Progress in Multi-turn Dialogue via Information Gaincited by 0