# Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems

arXiv: https://arxiv.org/abs/1911.01456 · HTML: https://mlsanity.com/paper/1911.01456/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2019.

## Built on by (most cited first, 30 of 36)

- [LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models](https://mlsanity.com/paper/2305.13711/) (2023)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations](https://mlsanity.com/paper/2203.09711/) (2022)
- [What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation](https://mlsanity.com/paper/2203.13927/) (2022)
- [Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges](https://mlsanity.com/paper/2203.10012/) (2022)
- [Approximating Online Human Evaluation of Social Chatbots with Prompting](https://mlsanity.com/paper/2304.05253/) (2023)
- [ACCENT: An Automatic Event Commonsense Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2305.07797/) (2023)
- [DiSCoL: Toward Engaging Dialogue Systems through Conversational Line Guided Response Generation](https://mlsanity.com/paper/2102.02191/) (2021)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Measuring Psychological Depth in Language Models](https://mlsanity.com/paper/2406.12680/) (2024)
- [From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment](https://mlsanity.com/paper/2505.16610/) (2025)
- [RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue](https://mlsanity.com/paper/2309.08156/) (2023)
- [EnDex: Evaluation of Dialogue Engagingness at Scale](https://mlsanity.com/paper/2210.12362/) (2022)
- [Modeling Performance in Open-Domain Dialogue with PARADISE](https://mlsanity.com/paper/2110.11164/) (2021)
- [MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue](https://mlsanity.com/paper/2206.09403/) (2022)
- [C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation](https://mlsanity.com/paper/2306.15245/) (2023)
- [FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows](https://mlsanity.com/paper/2202.06633/) (2022)
- [Improving Open-Domain Dialogue Evaluation with a Causal Inference Model](https://mlsanity.com/paper/2301.13372/) (2023)
- [A Transformer-based Response Evaluator for Open-Domain Spoken Conversation](https://mlsanity.com/paper/2302.04424/) (2023)
- [Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations](https://mlsanity.com/paper/2406.15000/) (2024)
- [User Response and Sentiment Prediction for Automatic Dialogue Evaluation](https://mlsanity.com/paper/2111.08808/) (2021)
- [Open-Domain Dialog Evaluation using Follow-Ups Likelihood](https://mlsanity.com/paper/2209.05185/) (2022)
- [PoE: a Panel of Experts for Generalized Automatic Dialogue Assessment](https://mlsanity.com/paper/2212.08992/) (2022)
- [Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs](https://mlsanity.com/paper/2306.03984/) (2023)
- [What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts](https://mlsanity.com/paper/2111.00572/) (2021)
- [Evaluate What You Can't Evaluate: Unassessable Quality for Generated Response](https://mlsanity.com/paper/2305.14658/) (2023)
- [Open-Domain Dialogue Quality Evaluation: Deriving Nugget-level Scores from Turn-level Scores](https://mlsanity.com/paper/2310.00410/) (2023)
- [BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation](https://mlsanity.com/paper/2501.10328/) (2025)
- [Development and Validation of Engagement and Rapport Scales for Evaluating User Experience in Multimodal Dialogue Systems](https://mlsanity.com/paper/2505.17075/) (2025)

## Abstract (arXiv)

User engagement is a critical metric for evaluating the quality of open-domain dialogue systems. Prior work has focused on conversation-level engagement by using heuristically constructed features such as the number of turns and the total time of the conversation. In this paper, we investigate the possibility and efficacy of estimating utterance-level engagement and define a novel metric, em predictive engagement, for automatic evaluation of open-domain dialogue systems. Our experiments demonstrate that (1) human annotators have high agreement on assessing utterance-level engagement scores; (2) conversation-level engagement scores can be predicted from properly aggregated utterance-level engagement scores. Furthermore, we show that the utterance-level engagement scores can be learned from data. These scores can improve automatic evaluation metrics for open-domain dialogue systems, as shown by correlation with human judgements. This suggests that predictive engagement can be used as a real-time feedback for training better dialogue models.