mlsanityData to 7 Oct 2026

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

arXiv 2401.07103 ↗First posted 13 Jan 2024cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

In the rapidly evolving domain of Natural Language Generation (NLG) evaluation, introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. This paper aims to provide a thorough overview of leveraging LLMs for NLG evaluation, a burgeoning area that lacks a systematic analysis. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. Our detailed exploration includes critically assessing various LLM-based methodologies, as well as comparing their strengths and limitations in evaluating NLG outputs. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 85 of its 127 references that are in the map. First posted on arXiv 13 Jan 2024.

Builds on

85 of 127 references.

References · 85

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2018Hierarchical Neural Story Generationcited by 1,182
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2015Teaching Machines to Read and Comprehendcited by 1,039
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  8. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  13. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  14. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  15. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  16. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  17. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  18. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  19. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  20. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  21. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  22. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  23. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  24. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  25. 2020Evaluation of Text Generation: A Surveycited by 275
  26. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  27. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  28. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  29. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  30. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  31. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  32. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  33. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  34. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  35. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  36. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  37. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  38. 2023Human-like Summarization Evaluation with ChatGPTcited by 126

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 16; 2026: 8.

37 papers.

Sort:

By area:

  1. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  2. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  3. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  4. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  5. 2024On the Challenges and Opportunities in Generative AIcited by 31
  6. 2024DRT: Deep Reasoning Translation via Long Chain-of-Thoughtcited by 28
  7. 2025BlenderGym: Benchmarking Foundational Model Systems for Graphics Editingcited by 27
  8. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  9. 2025Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generationcited by 26
  10. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  11. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  12. 2025OmniThink: Expanding Knowledge Boundaries in Machine Writing through Thinkingcited by 18
25 more
  1. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  2. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  3. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  4. 2025ProRefine: Inference-Time Prompt Refinement with Textual Feedbackcited by 8
  5. 2024Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Gamescited by 7
  6. 2024MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systemscited by 7
  7. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  8. 2024Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentationcited by 6
  9. 2025Expert Preference-based Evaluation of Automated Related Work Generationcited by 6
  10. 2024Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapycited by 5
  11. 2024A Comprehensive Evaluation on Event Reasoning of Large Language Modelscited by 4
  12. 2024Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?cited by 3
  13. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 3
  14. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  15. 2025EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generationcited by 2
  16. 2026Classroom AI: Large Language Models as Grade-Specific Teacherscited by 2
  17. 2026Reward Modeling for Scientific Writing Evaluationcited by 2
  18. 2025Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarizationcited by 1
  19. 2026ErrEval: Error-Aware Evaluation for Question Generation through Explicit Diagnosticscited by 1
  20. 2024Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluationcited by 0
  21. 2025A Conceptual Framework for AI Capability Evaluationscited by 0
  22. 2026From Physician Expertise to Clinical Agents: Preserving, Standardizing, and Scaling Physicians' Medical Expertise with Lightweight LLMcited by 0
  23. 2026XTC: Head-Aware Sampling by Excluding Top Choicescited by 0
  24. 2026Segment-Level Agentic Topic Modeling for Improved Data Exploration and Resource Efficiencycited by 0
  25. 2026Evaluating Whether LLMs Can Reliably Connect the DOTs?cited by 0