Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems
Not on the map. Why?
Abstract (arXiv)
Automation of dialogue system evaluation is a driving force for the efficient development of dialogue systems. This paper introduces the bipartite-play method, a dialogue collection method for automating dialogue system evaluation. It addresses the limitations of existing dialogue collection methods: (i) inability to compare with systems that are not publicly available, and (ii) vulnerability to cheating by intentionally selecting systems to be compared. Experimental results show that the automatic evaluation using the bipartite-play method mitigates these two drawbacks and correlates as strongly with human subjectivity as existing methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 19 Nov 2022.
Builds on
30 of 55 references.
References · 30
- 2017Attention Is All You Needcited by 59,932
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2020Recipes for building an open-domain chatbotcited by 593
- 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2020The Pushshift Reddit Datasetcited by 289
- 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
- 2021Internet-Augmented Dialogue Generationcited by 217
18 more
- 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
- 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
- 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
- 2017ParlAI: A Dialog Research Software Platformcited by 139
- 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
- 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
- 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
- 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
- 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
- 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
- 2018Image Chat: Engaging Grounded Conversationscited by 76
- 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
- 2019The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agentscited by 45
- 2019Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systemscited by 42
- 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
- 2020Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systemscited by 26
- 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
- 2019Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvementcited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.