# Towards a Metric for Automated Conversational Dialogue System Evaluation and Improvement

arXiv: https://arxiv.org/abs/1909.12066 · HTML: https://mlsanity.com/paper/1909.12066/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking](https://mlsanity.com/paper/2412.14140/) (2024)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [Probing the Robustness of Trained Metrics for Conversational Dialogue Systems](https://mlsanity.com/paper/2202.13887/) (2022)
- [On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems](https://mlsanity.com/paper/2104.06335/) (2021)
- [Bipartite-play Dialogue Collection for Practical Automatic Evaluation of Dialogue Systems](https://mlsanity.com/paper/2211.10596/) (2022)

## Abstract (arXiv)

We present "AutoJudge", an automated evaluation method for conversational dialogue systems. The method works by first generating dialogues based on self-talk, i.e. dialogue systems talking to itself. Then, it uses human ratings on these dialogues to train an automated judgement model. Our experiments show that AutoJudge correlates well with the human ratings and can be used to automatically evaluate dialogue systems, even in deployed systems. In a second part, we attempt to apply AutoJudge to improve existing systems. This works well for re-ranking a set of candidate utterances. However, our experiments show that AutoJudge cannot be applied as reward for reinforcement learning, although the metric can distinguish good from bad dialogues. We discuss potential reasons, but state here already that this is still an open question for further research.