mlsanityData to 7 Oct 2026

Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation

arXiv 2301.12004 ↗First posted 27 Jan 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Language models have steadily increased in size over the past few years. They achieve a high level of performance on various natural language processing (NLP) tasks such as question answering and summarization. Large language models (LLMs) have been used for generation and can now output human-like text. Due to this, there are other downstream tasks in the realm of dialog that can now harness the LLMs' language understanding capabilities. Dialog evaluation is one task that this paper will explore. It concentrates on prompting with LLMs: BLOOM, OPT, GPT-3, Flan-T5, InstructDial and TNLGv2. The paper shows that the choice of datasets used for training a model contributes to how well it performs on a task as well as on how the prompt should be structured. Specifically, the more diverse and relevant the group of datasets that a model is trained on, the better dialog evaluation performs. This paper also investigates how the number of examples in the prompt and the type of example selection used affect the model's performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 42 of its 47 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

42 of 47 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
30 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  7. 2019How Can We Know What Language Models Know?cited by 953
  8. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  10. 2020Recipes for building an open-domain chatbotcited by 593
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2019Defending Against Neural Fake Newscited by 553
  13. 2020The Pushshift Reddit Datasetcited by 289
  14. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  15. 2018A Simple Method for Commonsense Reasoningcited by 223
  16. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  17. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  18. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  19. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  20. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  21. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  22. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  23. 2019Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple Referencescited by 48
  24. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  25. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  26. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  27. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  28. 2020Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improvecited by 16
  29. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  30. 2021DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encodercited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 4; 2025: 1; 2026: 1.

12 papers.

  1. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  2. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  3. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  4. 2023CoAScore: Chain-of-Aspects Prompting for NLG Evaluationcited by 14
  5. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  6. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  7. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluationcited by 10
  8. 2024Leveraging LLMs for Dialogue Quality Measurementcited by 7
  9. 2024ECoh: Turn-level Coherence Evaluation for Multilingual Dialoguescited by 5
  10. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  11. 2024Education in the Era of Neurosymbolic AIcited by 4
  12. 2026Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreementcited by 2