mlsanityData to 14 Oct 2026

LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models

arXiv 2305.13711 ↗First posted 23 May 2023cs.CL cs.AIcited by 101

Not on the map. Why?

Abstract (arXiv)

We propose LLM-Eval, a unified multi-dimensional automatic evaluation method for open-domain conversations with large language models (LLMs). Existing evaluation methods often rely on human annotations, ground-truth responses, or multiple LLM prompts, which can be expensive and time-consuming. To address these issues, we design a single prompt-based evaluation method that leverages a unified evaluation schema to cover multiple dimensions of conversation quality in a single model call. We extensively evaluate the performance of LLM-Eval on various benchmark datasets, demonstrating its effectiveness, efficiency, and adaptability compared to state-of-the-art evaluation methods. Our analysis also highlights the importance of choosing suitable LLMs and decoding strategies for accurate evaluation results. LLM-Eval offers a versatile and robust solution for evaluating open-domain conversation systems, streamlining the evaluation process and providing consistent performance across diverse scenarios.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

28 of 41 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  8. 2023GPTScore: Evaluate as You Desirecited by 361
  9. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  10. 2015A Neural Conversational Modelcited by 196
  11. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  12. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
16 more
  1. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  2. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  3. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  4. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  5. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  6. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  7. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  8. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  9. 2020Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systemscited by 47
  10. 2021Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterancescited by 40
  11. 2019Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systemscited by 36
  12. 2021Automatic Evaluation and Moderation of Open-domain Dialogue Systemscited by 28
  13. 2017End-to-end Conversation Modeling Track in DSTC6cited by 12
  14. 2020An Evaluation Protocol for Generative Conversational Systemscited by 9
  15. 2022Interactive Evaluation of Dialog Track at DSTC9cited by 9
  16. 2022MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialoguecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 39; 2025: 27; 2026: 11.

101 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  5. 2023Challenges and Applications of Large Language Modelscited by 254
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  7. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  8. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  9. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  10. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  11. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  12. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 88
38 more
  1. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  2. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  3. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  4. 2023Integrating Large Language Model for Improved Causal Discoverycited by 54
  5. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  6. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 43
  7. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 42
  8. 2023AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answeringcited by 40
  9. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  10. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 37
  11. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  12. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  13. 2023Aligning Large Language Models with Human: A Surveycited by 32
  14. 2023Are Large Language Models Really Robust to Word-Level Perturbations?cited by 32
  15. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  16. 2023TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safetycited by 27
  17. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  18. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  19. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22
  20. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  21. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 19
  22. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  23. 2024Large Language Models Struggle with Unreasonability in Math Problemscited by 18
  24. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  25. 2024Are LLMs Correctly Integrated into Software Systems?cited by 15
  26. 2023LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversationcited by 14
  27. 2023Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suitecited by 14
  28. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  29. 2024GenAI Detection Tools, Adversarial Techniques and Implications for Inclusivity in Higher Educationcited by 14
  30. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  31. 2024Large Language Models are Capable of Offering Cognitive Reappraisal, if Guidedcited by 13
  32. 2024LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesiscited by 13
  33. 2024Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspectivecited by 11
  34. 2024PairEval: Open-domain Dialogue Evaluation with Pairwise Comparisoncited by 11
  35. 2024The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluatorscited by 11
  36. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 10
  37. 2024Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Toolscited by 9
  38. 2023PhoGPT: Generative Pre-training for Vietnamesecited by 8

50 most cited of 101 All 101 (JSON)