# Towards Best Experiment Design for Evaluating Dialogue System Output

arXiv: https://arxiv.org/abs/1909.10122 · HTML: https://mlsanity.com/paper/1909.10122/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 15 of 15)

- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [TellMeWhy: A Dataset for Answering Why-Questions in Narratives](https://mlsanity.com/paper/2106.06132/) (2021)
- [Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation](https://mlsanity.com/paper/2310.14424/) (2023)
- [Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries](https://mlsanity.com/paper/2109.09195/) (2021)
- [How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation](https://mlsanity.com/paper/2101.11298/) (2021)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [Affective Decoding for Empathetic Response Generation](https://mlsanity.com/paper/2108.08102/) (2021)
- [On the Effectiveness of Automated Metrics for Text Generation Systems](https://mlsanity.com/paper/2210.13025/) (2022)
- [Dynamic Human Evaluation for Relative Model Comparisons](https://mlsanity.com/paper/2112.08048/) (2021)
- [Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotations](https://mlsanity.com/paper/2408.09765/) (2024)
- [Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction](https://mlsanity.com/paper/2407.11857/) (2024)
- [Towards Detecting Inconsistencies in End-to-end Generated TODs](https://mlsanity.com/paper/2607.09338/) (2026)

## Abstract (arXiv)

To overcome the limitations of automated metrics (e.g. BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence. While it has been demonstrated that human judgments can suffer from the inconsistency of ratings, extant research has also found that the design of the evaluation task affects the consistency and quality of human judgments. We conduct a between-subjects study to understand the impact of four experiment conditions on human ratings of dialogue system output. In addition to discrete and continuous scale ratings, we also experiment with a novel application of Best-Worst scaling to dialogue evaluation. Through our systematic study with 40 crowdsourced workers in each task, we find that using continuous scales achieves more consistent ratings than Likert scale or ranking-based experiment design. Additionally, we find that factors such as time taken to complete the task and no prior experience of participating in similar studies of rating dialogue system output positively impact consistency and agreement amongst raters