mlsanityData to 7 Oct 2026

How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation

arXiv 2305.14533 ↗First posted 23 May 2023cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We release MMSMR, a Massively Multi-System MultiReference dataset to enable future work on metrics and evaluation for dialog. Automatic metrics for dialogue evaluation should be robust proxies for human judgments; however, the verification of robustness is currently far from satisfactory. To quantify the robustness correlation and understand what is necessary in a test set, we create and release an 8-reference dialog dataset by extending single-reference evaluation sets and introduce this new language learning conversation dataset. We then train 1750 systems and evaluate them on our novel test set and the DailyDialog dataset. We release the novel test set, and model hyper parameters, inference outputs, and metric scores for each system on a variety of datasets.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2015A Neural Conversational Modelcited by 195
  11. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  12. 2017ParlAI: A Dialog Research Software Platformcited by 139
10 more
  1. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  2. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  3. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  4. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  5. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  6. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  7. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  8. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  9. 2015deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targetscited by 21
  10. 2021Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023KEYword based Sampling (KEYS) for Large Language Modelscited by 0