mlsanityData to 14 Oct 2026

ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues

arXiv 2407.11660 ↗First posted 16 Jul 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence). GenResCoh is a novel LLM generated dataset comprising over 130k negative and positive responses and accompanying explanations seeded from XDailyDialog and XPersona covering English, French, German, Italian, and Chinese. Leveraging GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators trained to assess response coherence across multiple languages. Experimental results demonstrate that ECoh achieves multilingual detection capabilities superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based on a much smaller architecture. Furthermore, the explanations provided by ECoh closely align in terms of quality with those generated by the teacher model.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

29 of 45 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  7. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  8. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  9. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  10. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  11. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  12. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
17 more
  1. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  2. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  3. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  4. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  5. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  6. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  7. 2023Leveraging GPT-4 for Automatic Translation Post-Editingcited by 56
  8. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  9. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  10. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  11. 2020XPersona: Evaluating Multilingual Personalized Chatbotcited by 36
  12. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
  13. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  14. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  15. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  16. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  17. 2023Understanding the Effectiveness of Very Large Language Models on Dialog Evaluationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  2. 2025Evaluation and Facilitation of Online Discussions in the LLM Era: A Surveycited by 9
  3. 2025Joint Modeling of Entities and Discourse Relations for Coherence Assessmentcited by 2
  4. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
  5. 2026SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios
  2. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  3. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?
  4. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  5. 2026Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation
  6. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
  7. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation
  8. 2026WavBench: Benchmarking Reasoning, Colloquialism, and Paralinguistics for End-to-End Spoken Dialogue Models