# Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation

arXiv: https://arxiv.org/abs/2308.16797 · HTML: https://mlsanity.com/paper/2308.16797/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems](https://mlsanity.com/paper/1701.03079/) (2017)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems](https://mlsanity.com/paper/2010.03994/) (2020)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Unbabel's Participation in the WMT20 Metrics Shared Task](https://mlsanity.com/paper/2010.15535/) (2020)
- [Learning an Unreferenced Metric for Online Dialogue Evaluation](https://mlsanity.com/paper/2005.00583/) (2020)
- [Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2011.00483/) (2020)
- [Designing Precise and Robust Dialogue Response Evaluators](https://mlsanity.com/paper/2004.04908/) (2020)
- [Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation](https://mlsanity.com/paper/2301.12004/) (2023)
- [Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4](https://mlsanity.com/paper/2306.12794/) (2023)
- [EnDex: Evaluation of Dialogue Engagingness at Scale](https://mlsanity.com/paper/2210.12362/) (2022)
- [MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue](https://mlsanity.com/paper/2206.09403/) (2022)

## Built on by (most cited first, 19 of 19)

- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Are LLM-based Evaluators Confusing NLG Quality Criteria?](https://mlsanity.com/paper/2402.12055/) (2024)
- [A survey of dynamic graph neural networks](https://mlsanity.com/paper/2404.18211/) (2024)
- [Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set](https://mlsanity.com/paper/2312.12624/) (2023)
- [Cohesive Conversations: Enhancing Authenticity in Multi-Agent Simulated Dialogues](https://mlsanity.com/paper/2407.09897/) (2024)
- [On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation](https://mlsanity.com/paper/2407.03841/) (2024)
- [Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs](https://mlsanity.com/paper/2408.10902/) (2024)
- [Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Task](https://mlsanity.com/paper/2311.00686/) (2023)
- [Multilingual Prompt Engineering in Large Language Models: A Survey Across NLP Tasks](https://mlsanity.com/paper/2505.11665/) (2025)
- [ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues](https://mlsanity.com/paper/2407.11660/) (2024)
- [Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators](https://mlsanity.com/paper/2503.04360/) (2025)
- [From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentary](https://mlsanity.com/paper/2506.17294/) (2025)
- [CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments](https://mlsanity.com/paper/2603.28569/) (2026)
- [Grammar Control in Dialogue Response Generation for Language Learning Chatbots](https://mlsanity.com/paper/2502.07544/) (2025)
- [When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue](https://mlsanity.com/paper/2510.19186/) (2025)
- [ConvoCache: Smart Re-Use of Chatbot Responses](https://mlsanity.com/paper/2406.18133/) (2024)
- [Culturally-Grounded Governance for Multilingual Language Models: Rights, Data Boundaries, and Accountable AI Design](https://mlsanity.com/paper/2602.00497/) (2026)
- [Evaluating Zero-Shot and One-Shot Adaptation of Small Language Models in Leader-Follower Interaction](https://mlsanity.com/paper/2602.23312/) (2026)

## Abstract (arXiv)

Despite significant research effort in the development of automatic dialogue evaluation metrics, little thought is given to evaluating dialogues other than in English. At the same time, ensuring metrics are invariant to semantically similar responses is also an overlooked topic. In order to achieve the desired properties of robustness and multilinguality for dialogue evaluation metrics, we propose a novel framework that takes advantage of the strengths of current evaluation models with the newly-established paradigm of prompting Large Language Models (LLMs). Empirical results show our framework achieves state of the art results in terms of mean Spearman correlation scores across several benchmarks and ranks first place on both the Robust and Multilingual tasks of the DSTC11 Track 4 "Automatic Evaluation Metrics for Open-Domain Dialogue Systems", proving the evaluation capabilities of prompted LLMs.