mlsanityData to 7 Oct 2026

LLM-based NLG Evaluation: Current Status and Challenges

arXiv 2402.01383 ↗First posted 2 Feb 2024cs.CLcited by 71

Not on the map. Why?

Abstract (arXiv)

Evaluating natural language generation (NLG) is a vital but challenging problem in natural language processing. Traditional evaluation metrics mainly capturing content (e.g. n-gram) overlap between system outputs and references are far from satisfactory, and large language models (LLMs) such as ChatGPT have demonstrated great potential in NLG evaluation in recent years. Various automatic evaluation methods based on LLMs have been proposed, including metrics derived from LLMs, prompting LLMs, fine-tuning LLMs, and human-LLM collaborative evaluation. In this survey, we first give a taxonomy of LLM-based NLG evaluation methods, and discuss their pros and cons, respectively. Lastly, we discuss several open problems in this area and point out future research directions.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 100 of its 135 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

100 of 135 references.

References · 100

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2016Concrete Problems in AI Safetycited by 1,582
38 more
  1. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  5. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  6. 2023Large Language Modelscited by 774
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  13. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  14. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  15. 2022Self-critiquing models for assisting human evaluatorscited by 372
  16. 2023GPTScore: Evaluate as You Desirecited by 357
  17. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  18. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  19. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  20. 2022Teaching language models to support answers with verified quotescited by 290
  21. 2020Evaluation of Text Generation: A Surveycited by 275
  22. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  23. 2017Challenges in Data-to-Document Generationcited by 226
  24. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  25. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  26. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  27. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  28. 2023Generative Judge for Evaluating Alignmentcited by 163
  29. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  30. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  31. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  32. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  33. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  34. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  35. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  36. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  37. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  38. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 32; 2026: 15.

71 papers.

Sort:

By area:

  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  2. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  3. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  4. 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 46
  5. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  6. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  7. 2025Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generationcited by 26
  8. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  9. 2025Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunitiescited by 21
  10. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  11. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
  12. 2024Natural Language Reinforcement Learningcited by 17
38 more
  1. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  2. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  3. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  4. 2025A Dataset for Addressing Patient's Information Needs related to Clinical Course of Hospitalizationcited by 13
  5. 2026Mano: Restriking Manifold Optimization for LLM Trainingcited by 12
  6. 2024LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Datacited by 10
  7. 2024Leveraging Large Language Models for Integrated Satellite-Aerial-Terrestrial Networks: Recent Advances and Future Directionscited by 10
  8. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  9. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 9
  10. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 9
  11. 2024A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutionscited by 8
  12. 2024An Auditing Test To Detect Behavioral Shift in Language Modelscited by 8
  13. 2024Automated Review Generation Method Based on Large Language Modelscited by 7
  14. 2025Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Datacited by 7
  15. 2025LLM-Driven E-Commerce Marketing Content Optimization: Balancing Creativity and Conversioncited by 7
  16. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  17. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarizationcited by 6
  18. 2024Towards Logically Sound Natural Language Reasoning with Logic-Enhanced Language Model Agentscited by 5
  19. 2025A Comprehensive LLM-powered Framework for Driving Intelligence Evaluationcited by 5
  20. 2025When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?cited by 5
  21. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  22. 2024Large Language Models Meet User Interfaces: The Case of Provisioning Feedbackcited by 4
  23. 2024Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunitiescited by 4
  24. 2024Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesiscited by 4
  25. 2024Natural Language Reinforcement Learningcited by 4
  26. 2024Reasoning over User Preferences: Knowledge Graph-Augmented LLMs for Explainable Conversational Recommendationscited by 4
  27. 2025ConSens: Assessing context grounding in open-book question answeringcited by 4
  28. 2024Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarizationcited by 3
  29. 2025Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleavingcited by 3
  30. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Promptingcited by 3
  31. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  32. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  33. 2025Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Modelscited by 2
  34. 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Referencecited by 2
  35. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  36. 2024Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairscited by 1
  37. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  38. 2025Reasoning in Action: MCTS-Driven Knowledge Retrieval for Large Language Modelscited by 1

50 most cited of 71 All 71 (JSON)