SelF-Eval: Self-supervised Fine-grained Dialogue Evaluation
Not on the map. Why?
Abstract (arXiv)
This paper introduces a novel Self-supervised Fine-grained Dialogue Evaluation framework (SelF-Eval). The core idea is to model the correlation between turn quality and the entire dialogue quality. We first propose a novel automatic data construction method that can automatically assign fine-grained scores for arbitrarily dialogue data. Then we train SelF-Eval with a multi-level contrastive learning schema which helps to distinguish different score levels. Experimental results on multiple benchmarks show that SelF-Eval is highly consistent with human evaluations and better than the state-of-the-art models. We give a detailed analysis of the experiments in this paper. Our code is available on GitHub.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 17 Aug 2022.
Builds on
34 of 49 references.
References · 34
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
- 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2017Why We Need New Evaluation Metrics for NLGcited by 232
22 more
- 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
- 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
- 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
- 2018Neural Approaches to Conversational AIcited by 164
- 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
- 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
- 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
- 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
- 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
- 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
- 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
- 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
- 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
- 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
- 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
- 2020Overview of the Ninth Dialog System Technology Challenge: DSTC9cited by 49
- 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
- 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
- 2019Generating Persona Consistent Dialogues by Exploiting Natural Language Inferencecited by 33
- 2018Better Conversations by Modeling,Filtering,and Optimizing for Coherence and Diversitycited by 27
- 2021Towards Quantifiable Dialogue Coherence Evaluationcited by 21
- 2021DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialoguescited by 7
Built on by
Citing papers per year: 2023: 2.
2 papers.