mlsanityData to 7 Oct 2026

Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs

arXiv 2408.10902 ↗First posted 20 Aug 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Although human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also extended to dialogue. However, most frameworks leverage benchmarks that assess older chatbots on aspects such as fluency and relevance, which are not reflective of the challenges associated with contemporary models. In fact, a qualitative analysis on Soda, a GPT-3.5 generated dialogue dataset, suggests that current chatbots may exhibit several recurring issues related to coherence and commonsense knowledge, but generally produce highly fluent and relevant responses. Noting the aforementioned limitations, this paper introduces Soda-Eval, an annotated dataset based on Soda that covers over 120K turn-level assessments across 10K dialogues, where the annotations were generated by GPT-4. Using Soda-Eval as a benchmark, we then study the performance of several open-access instruction-tuned LLMs, finding that dialogue evaluation remains challenging. Fine-tuning these models improves performance over few-shot inferences, both in terms of correlation and explanation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 53 of its 83 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

53 of 83 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  12. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
38 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  3. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2020Recipes for building an open-domain chatbotcited by 593
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  11. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  13. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  14. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  15. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  16. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
  17. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  18. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  19. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  20. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  21. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  22. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  23. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  24. 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
  25. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  26. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  27. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  28. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  29. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  30. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  31. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  32. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  33. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  34. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  35. 2020Emora: An Inquisitive Social Chatbot Who Cares For Youcited by 16
  36. 2023Leveraging Large Language Models for Automated Dialogue Analysiscited by 14
  37. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  38. 2023Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4cited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 1.

10 papers.

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  3. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  4. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  5. 2025PersonaTwin: A Multi-Tier Prompt Conditioning Framework for Generating and Evaluating Personalized Digital Twinscited by 5
  6. 2025A Dual-Perspective NLG Meta-Evaluation Framework with Automatic Benchmark and Better Interpretabilitycited by 1
  7. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  8. 2025Overview of Dialog System Evaluation Track: Dimensionality, Language, Culture and Safety at DSTC 12cited by 1
  9. 2025Towards Automated Error Discovery: A Study in Conversational AIcited by 0
  10. 2026Controllable Spoken Dialogue Generation: An LLM-Driven Grading System for K-12 Non-Native English Learnerscited by 0