mlsanityData to 7 Oct 2026

DynaEval: Unifying Turn and Dialogue Level Evaluation

arXiv 2106.01112 ↗First posted 2 Jun 2021cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

A dialogue is essentially a multi-turn interaction among interlocutors. Effective evaluation metrics should reflect the dynamics of such interaction. Existing automatic metrics are focused very much on the turn-level quality, while ignoring such dynamics. To this end, we propose DynaEval, a unified automatic evaluation framework which is not only capable of performing turn-level evaluation, but also holistically considers the quality of the entire dialogue. In DynaEval, the graph convolutional network (GCN) is adopted to model a dialogue in totality, where the graph nodes denote each individual utterance and the edges represent the dependency between pairs of utterances. A contrastive loss is then applied to distinguish well-formed dialogues from carefully constructed negative samples. Experiments show that DynaEval significantly outperforms the state-of-the-art dialogue coherence model, and correlates strongly with human judgements across multiple dialogue evaluation aspects at both turn and dialogue level.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 37 of its 64 references that are in the map. First posted on arXiv 2 Jun 2021.

Builds on

37 of 64 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017Graph Attention Networkscited by 8,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  10. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
25 more
  1. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  2. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  3. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  4. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  5. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  6. 2018Dialogue Natural Language Inferencecited by 135
  7. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  8. 2019Knowledge-Enriched Transformer for Emotion Detection in Textual Conversationscited by 92
  9. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  10. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  11. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  12. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  13. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  14. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  15. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  16. 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
  17. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  18. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  19. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  20. 2020Co-GAT: A Co-Interactive Graph Attention Network for Joint Dialog Act Recognition and Sentiment Classificationcited by 29
  21. 2018Better Conversations by Modeling,Filtering,and Optimizing for Coherence and Diversitycited by 27
  22. 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
  23. 2020Is this Dialogue Coherent? Learning from Dialogue Acts and Entitiescited by 11
  24. 2018Coherence Models for Dialoguecited by 9
  25. 2020Evaluating Dialogue Generation Systems via Response Selectioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 21; 2024: 10; 2025: 6; 2026: 3.

59 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  2. 2023GPTScore: Evaluate as You Desirecited by 357
  3. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 139
  4. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  5. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  6. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  7. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  8. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  9. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  10. 2023X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspectscited by 28
  11. 2025Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learningcited by 27
  12. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
38 more
  1. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 22
  2. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  3. 2022What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluationcited by 18
  4. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 18
  5. 2022Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policycited by 17
  6. 2023BatchEval: Towards Human-like Text Evaluationcited by 16
  7. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  8. 2023Leveraging Large Language Models for Automated Dialogue Analysiscited by 14
  9. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  10. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  11. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  12. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  13. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  14. 2021Exploring Dense Retrieval for Dialogue Response Selectioncited by 11
  15. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  16. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  17. 2023Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sourcescited by 9
  18. 2023RoleCraft-GLM: Advancing Personalized Role-Playing in Large Language Modelscited by 9
  19. 2025Evaluation and Facilitation of Online Discussions in the LLM Era: A Surveycited by 8
  20. 2023RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialoguecited by 7
  21. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  22. 2023C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluationcited by 5
  23. 2024CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systemscited by 5
  24. 2024ECoh: Turn-level Coherence Evaluation for Multilingual Dialoguescited by 5
  25. 2021Investigating the Impact of Pre-trained Language Models on Dialog Evaluationcited by 4
  26. 2022FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flowscited by 4
  27. 2023Improving Open-Domain Dialogue Evaluation with a Causal Inference Modelcited by 4
  28. 2023A Transformer-based Response Evaluator for Open-Domain Spoken Conversationcited by 4
  29. 2024MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphscited by 4
  30. 2024Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Modelscited by 4
  31. 2024Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challengescited by 4
  32. 2022Open-Domain Dialog Evaluation using Follow-Ups Likelihoodcited by 3
  33. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  34. 2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3
  35. 2023Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogscited by 3
  36. 2024Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editingcited by 3
  37. 2025Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judgescited by 3
  38. 2022SelF-Eval: Self-supervised Fine-grained Dialogue Evaluationcited by 2

50 most cited of 59 All 59 (JSON)