Three Ways of Using Large Language Models to Evaluate Chat
Not on the map. Why?
Abstract (arXiv)
This paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition. We present three different approaches to predicting turn-level qualities of chatbot responses based on large language models (LLMs). We report improvement over the baseline using dynamic few-shot examples from a vector store for the prompts for ChatGPT. We also analyze the performance of the other two approaches and report needed improvements for future work. We developed the three systems over just two weeks, showing the potential of LLMs for this task. An ablation study conducted after the challenge deadline shows that the new Llama 2 models are closing the performance gap between ChatGPT and open-source LLMs. However, we find that the Llama 2 models do not benefit from few-shot examples in the same way as ChatGPT.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 12 Aug 2023.
Builds on
27 of 35 references.
References · 27
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2017Billion-scale similarity search with GPUscited by 2,477
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
15 more
- 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
- 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
- 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
- 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
- 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
- 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
- 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
- 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
- 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
- 2020Learning an Unreferenced Metric for Online Dialogue Evaluationcited by 52
- 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
- 2019Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responsescited by 22
- 2022Statistical Properties of the log-cosh Loss Function Used in Machine Learningcited by 15
- 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13
- 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10
Built on by
Citing papers per year: 2024: 2.
2 papers.