mlsanityData to 14 Oct 2026

Investigating the Impact of Pre-trained Language Models on Dialog Evaluation

arXiv 2110.01895 ↗First posted 5 Oct 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, there is a surge of interest in applying pre-trained language models (Pr-LM) in automatic open-domain dialog evaluation. Pr-LMs offer a promising direction for addressing the multi-domain evaluation challenge. Yet, the impact of different Pr-LMs on the performance of automatic metrics is not well-understood. This paper examines 8 different Pr-LMs and studies their impact on three typical automatic dialog evaluation metrics across three different dialog evaluation benchmarks. Specifically, we analyze how the choice of Pr-LMs affects the performance of automatic metrics. Extensive correlation analyses on each of the metrics are performed to assess the effects of different Pr-LMs along various axes, including pre-training objectives, dialog evaluation criteria, model size, and cross-dataset robustness. This study serves as the first comprehensive assessment of the effects of different Pr-LMs on automatic dialog evaluation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

21 of 28 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
9 more
  1. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  2. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  3. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  4. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  5. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  6. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  7. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  8. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  9. 2020Is this Dialogue Coherent? Learning from Dialogue Acts and Entitiescited by 11

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 1; 2023: 1.

4 papers.

  1. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  2. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
  3. 2023RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamicscited by 6
  4. 2022PoE: a Panel of Experts for Generalized Automatic Dialogue Assessmentcited by 3