mlsanityData to 7 Oct 2026

Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References

arXiv 1907.10568 ↗First posted 24 Jul 2019cs.CLcited by 48

Not on the map. Why?

Abstract (arXiv)

The aim of this paper is to mitigate the shortcomings of automatic evaluation of open-domain dialog systems through multi-reference evaluation. Existing metrics have been shown to correlate poorly with human judgement, particularly in open-domain dialog. One alternative is to collect human annotations for evaluation, which can be expensive and time consuming. To demonstrate the effectiveness of multi-reference evaluation, we augment the test set of DailyDialog with multiple references. A series of experiments show that the use of multiple references results in improved correlation between several automatic metrics and human judgement for both the quality and the diversity of system output.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 13; 2023: 11; 2024: 8; 2025: 4.

48 papers.

Sort:

By area:

  1. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  2. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  3. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  4. 2022PAL: Persona-Augmented Emotional Support Conversation Generationcited by 39
  5. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  6. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  7. 2021Investigating Robustness of Dialog Models to Popular Figurative Language Constructscited by 19
  8. 2021Ditch the Gold Standard: Re-evaluating Conversational Question Answeringcited by 19
  9. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  10. 2024Long-Form Answers to Visual Questions from Blind and Low Vision Peoplecited by 18
  11. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  12. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
36 more
  1. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  2. 2021ParaQA: A Question Answering Dataset with Paraphrase Responses for Single-Turn Conversationcited by 12
  3. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12
  4. 2021How Should Agents Ask Questions For Situated Learning? An Annotated Dialogue Corpuscited by 11
  5. 2023Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying Referencescited by 11
  6. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
  7. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  8. 2021HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversationscited by 9
  9. 2022Interactive Evaluation of Dialog Track at DSTC9cited by 9
  10. 2023DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversationscited by 9
  11. 2023Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sourcescited by 9
  12. 2021Language Model Augmented Relevance Scorecited by 7
  13. 2025HUME: Measuring the Human-Model Performance Gap in Text Embedding Taskscited by 7
  14. 2021Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentationcited by 6
  15. 2022Distribution Aware Metrics for Conditional Natural Language Generationcited by 6
  16. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5
  17. 2022A Systematic Evaluation of Response Selection for Open Domain Dialoguecited by 4
  18. 2023Psychological Metrics for Dialog System Evaluationcited by 4
  19. 2024MP2D: An Automated Topic Shift Dialogue Generation Framework Leveraging Knowledge Graphscited by 4
  20. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  21. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3
  22. 2023Attribute Controlled Dialogue Promptingcited by 3
  23. 2024Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrasescited by 3
  24. 2025Modeling the One-to-Many Property in Open-Domain Dialogue with LLMscited by 3
  25. 2021How to Evaluate Your Dialogue Models: A Review of Approachescited by 2
  26. 2022DialoGen: Generalized Long-Range Context Representation for Dialogue Systemscited by 2
  27. 2022There Is No Standard Answer: Knowledge-Grounded Dialogue Generation with Adversarial Activated Multi-Reference Learningcited by 2
  28. 2021Enhancing Self-Disclosure In Neural Dialog Models By Candidate Re-rankingcited by 1
  29. 2023How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluationcited by 1
  30. 2025BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generationcited by 1
  31. 2022Should we tweet this? Generative response modeling for predicting reception of public health messaging on Twittercited by 0
  32. 2023X-ReCoSa: Multi-Scale Context Aggregation For Multi-Turn Dialogue Generationcited by 0
  33. 2023DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue Assessmentcited by 0
  34. 2024Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systemscited by 0
  35. 2024Language Portability Strategies for Open-domain Dialogue with Pre-trained Language Models from High to Low Resource Languagescited by 0
  36. 2024Dialogue You Can Trust: Human and AI Perspectives on Generated Conversationscited by 0