mlsanityData to 14 Oct 2026

ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons

arXiv 1909.03087 ↗First posted 6 Sep 2019cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

While dialogue remains an important end-goal of natural language research, the difficulty of evaluation is an oft-quoted reason why it remains troublesome to make real progress towards its solution. Evaluation difficulties are actually two-fold: not only do automatic metrics not correlate well with human judgments, but also human judgments themselves are in fact difficult to measure. The two most used human judgment tests, single-turn pairwise evaluation and multi-turn Likert scores, both have serious flaws as we discuss in this work. We instead provide a novel procedure involving comparing two full dialogues, where a human judge is asked to pay attention to only one speaker within each, and make a pairwise judgment. The questions themselves are optimized to maximize the robustness of judgments across different annotators, resulting in better tests. We also show how these tests work in self-play model chat setups, resulting in faster, cheaper tests. We hope these tests become the de facto standard, and will release open-source code to that end.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 18; 2022: 24; 2023: 17; 2024: 15; 2025: 4; 2026: 3.

83 papers.

Sort:

By area:

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  5. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  6. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  7. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 78
  8. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 74
  9. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  10. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  11. 2022Keep Me Updated! Memory Management in Long-term Conversationscited by 63
  12. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
38 more
  1. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  2. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  3. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  4. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 51
  5. 2021SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failurescited by 37
  6. 2024Large Language Models in Mental Health Care: a Scoping Reviewcited by 34
  7. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialoguecited by 32
  8. 2022Achieving Reliable Human Assessment of Open-Domain Dialogue Systemscited by 31
  9. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  10. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31
  11. 2023Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Modelscited by 31
  12. 2023The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialoguescited by 27
  13. 2022A Literature Survey of Recent Advances in Chatbotscited by 26
  14. 2022Evaluating Mixed-initiative Conversational Search Systems via User Simulationcited by 25
  15. 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
  16. 2023Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curationcited by 23
  17. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  18. 2024Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLMcited by 21
  19. 2021Assessing Political Prudence of Open-domain Chatbotscited by 20
  20. 2024Label-Efficient Model Selection for Text Generationcited by 19
  21. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 19
  22. 2023Large Language Models Perform Diagnostic Reasoningcited by 18
  23. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  24. 2022PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentationcited by 14
  25. 2022Report from the NSF Future Directions Workshop on Automatic Evaluation of Dialog: Research Directions and Challengescited by 13
  26. 2022BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasetscited by 13
  27. 2023Mind the Gap Between Conversations for Improved Long-Term Dialogue Generationcited by 13
  28. 2024Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactionscited by 13
  29. 2024LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluationcited by 13
  30. 2021Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistencycited by 12
  31. 2022KETOD: Knowledge-Enriched Task-Oriented Dialoguecited by 12
  32. 2023Can Language Model Moderators Improve the Health of Online Discourse?cited by 12
  33. 2021LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcingcited by 11
  34. 2023Approximating Online Human Evaluation of Social Chatbots with Promptingcited by 11
  35. 2022Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisonscited by 10
  36. 2021HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversationscited by 9
  37. 2022Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systemscited by 9
  38. 2021DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialoguescited by 7

50 most cited of 83 All 83 (JSON)