mlsanityData to 7 Oct 2026

Automatic Evaluation and Moderation of Open-domain Dialogue Systems

arXiv 2111.02110 ↗First posted 3 Nov 2021cs.CL cs.HCcited by 28

Not on the map. Why?

Abstract (arXiv)

The development of Open-Domain Dialogue Systems (ODS)is a trending topic due to the large number of research challenges, large societal and business impact, and advances in the underlying technology. However, the development of these kinds of systems requires two important characteristics:1) automatic evaluation mechanisms that show high correlations with human judgements across multiple dialogue evaluation aspects (with explainable features for providing constructive and explicit feedback on the quality of generative models' responses for quick development and deployment)and 2) mechanisms that can help to control chatbot responses,while avoiding toxicity and employing intelligent ways to handle toxic user comments and keeping interaction flow and engagement. This track at the 10th Dialogue System Technology Challenge (DSTC10) is part of the ongoing effort to promote scalable and toxic-free ODS. This paper describes the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

35 of 58 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  8. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  10. 2020Recipes for building an open-domain chatbotcited by 593
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
23 more
  1. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  2. 2021Internet-Augmented Dialogue Generationcited by 217
  3. 2015A Neural Conversational Modelcited by 195
  4. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  5. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  6. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  7. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  8. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  9. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  10. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  11. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  12. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  13. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  14. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  15. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  16. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  17. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  18. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  19. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  20. 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
  21. 2017End-to-end Conversation Modeling Track in DSTC6cited by 12
  22. 2020An Evaluation Protocol for Generative Conversational Systemscited by 9
  23. 2021Investigating the Impact of Pre-trained Language Models on Dialog Evaluationcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 11; 2024: 7; 2025: 3.

28 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  3. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  4. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  5. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  6. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  7. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  8. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  9. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  10. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  11. 2023The DSA Transparency Database: Auditing Self-reported Moderation Actions by Social Mediacited by 10
  12. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
16 more
  1. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  2. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5
  3. 2023Psychological Metrics for Dialog System Evaluationcited by 4
  4. 2023Dialogue Quality and Emotion Annotations for Customer Support Conversationscited by 4
  5. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  6. 2024ASTPrompter: Preference-Aligned Automated Language Model Red-Teaming to Generate Low-Perplexity Unsafe Promptscited by 3
  7. 2025Toxicity in Online Platforms and AI Systems: A Survey of Needs, Challenges, Mitigations, and Future Directionscited by 3
  8. 2023Evaluate What You Can't Evaluate: Unassessable Quality for Generated Responsecited by 2
  9. 2024An Analysis of User Behaviors for Objectively Evaluating Spoken Dialogue Systemscited by 2
  10. 2023How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluationcited by 1
  11. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  12. 2025Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12cited by 1
  13. 2022Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metriccited by 0
  14. 2023Towards Objective Evaluation of Socially-Situated Conversational Robots: Assessing Human-Likeness through Multimodal User Behaviorscited by 0
  15. 2023Towards Multilingual Automatic Dialogue Evaluationcited by 0
  16. 2024A Unifying Information-theoretic Perspective on Evaluating Generative Modelscited by 0