mlsanityData to 7 Oct 2026

Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challenges

arXiv 2203.10012 ↗First posted 18 Mar 2022cs.CL cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

This is a report on the NSF Future Directions Workshop on Automatic Evaluation of Dialog. The workshop explored the current state of the art along with its limitations and suggested promising directions for future work in this important and very rapidly changing area of research.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 63 of its 106 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

63 of 106 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  8. 2020Recipes for building an open-domain chatbotcited by 593
  9. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  10. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  11. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  12. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
38 more
  1. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  2. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  3. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  4. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  5. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  6. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  7. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  8. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  9. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  10. 2016Learning End-to-End Goal-Oriented Dialogcited by 104
  11. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  12. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  13. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  14. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  15. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  16. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  17. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  18. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  19. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  20. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  21. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  22. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  23. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  24. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  25. 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
  26. 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
  27. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  28. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  29. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  30. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  31. 2016A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systemscited by 39
  32. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  33. 2019Incremental Transformer with Deliberation Decoder for Document Grounded Conversationscited by 35
  34. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  35. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  36. 2016On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systemscited by 28
  37. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  38. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 8.

13 papers.

Sort:

  1. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  2. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  3. 2023Mirages: On Anthropomorphism in Dialogue Systemscited by 30
  4. 2023DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AIcited by 21
  5. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  6. 2023InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanationscited by 11
  7. 2022TaTa: A Multilingual Table-to-Text Dataset for African Languagescited by 10
  8. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  9. 2023MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with Reliability Evaluationcited by 10
  10. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
  11. 2023The Open-domain Paradox for Chatbots: Common Ground as the Basis for Human-like Dialoguecited by 4
  12. 2022Dialogue Evaluation with Offline Reinforcement Learningcited by 3
1 more
  1. 2022Open-Domain Dialog Evaluation using Follow-Ups Likelihoodcited by 3