mlsanityData to 14 Oct 2026

Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4

arXiv 2306.12794 ↗First posted 22 Jun 2023cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

The advent and fast development of neural networks have revolutionized the research on dialogue systems and subsequently have triggered various challenges regarding their automatic evaluation. Automatic evaluation of open-domain dialogue systems as an open challenge has been the center of the attention of many researchers. Despite the consistent efforts to improve automatic metrics' correlations with human evaluation, there have been very few attempts to assess their robustness over multiple domains and dimensions. Also, their focus is mainly on the English language. All of these challenges prompt the development of automatic evaluation metrics that are reliable in various domains, dimensions, and languages. This track in the 11th Dialogue System Technology Challenge (DSTC11) is part of the ongoing effort to promote robust and multilingual automatic evaluation metrics. This article describes the datasets and baselines provided to participants and discusses the submission and result details of the two proposed subtasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 41 of its 71 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

41 of 71 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  8. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  9. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
29 more
  1. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  2. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  3. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  4. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  5. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  6. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  7. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  8. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  9. 2018A Dataset for Document Grounded Conversationscited by 104
  10. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  11. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  12. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  13. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  14. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  15. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  16. 2018Modeling Multi-turn Conversation with Deep Utterance Aggregationcited by 65
  17. 2020KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversationcited by 59
  18. 2018Towards Exploiting Background Knowledge for Building Conversation Systemscited by 57
  19. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  20. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  21. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  22. 2021PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generationcited by 37
  23. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  24. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  25. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  26. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  27. 2020Is this Dialogue Coherent? Learning from Dialogue Acts and Entitiescited by 11
  28. 2020Sentimental LIAR: Extended Corpus and Deep Learning Models for Fake Claim Classificationcited by 7
  29. 2022EnDex: Evaluation of Dialogue Engagingness at Scalecited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 4; 2026: 1.

10 papers.

  1. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  2. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  3. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  4. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  5. 2023Three Ways of Using Large Language Models to Evaluate Chatcited by 2
  6. 2025Grammar Control in Dialogue Response Generation for Language Learning Chatbotscited by 2
  7. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  8. 2025Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12cited by 1
  9. 2024Dialogue You Can Trust: Human and AI Perspectives on Generated Conversationscited by 0
  10. 2026PMIScore: An Unsupervised Approach to Quantify Dialogue Engagementcited by 0