mlsanityData to 7 Oct 2026

Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization

arXiv 2305.13091 ↗First posted 22 May 2023cs.CLcited by 70

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

With the recent undeniable advancement in reasoning abilities in large language models (LLMs) like ChatGPT and GPT-4, there is a growing trend for using LLMs on various tasks. One area where LLMs can be employed is as an alternative evaluation metric for complex generative tasks, which generally demands expensive human judges to complement the traditional automatic metrics for various evaluation dimensions such as fluency and consistency. In this work, we conduct extensive analysis to investigate the stability and reliability of LLMs as automatic evaluators for abstractive summarization. We found that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements due to significant limitations. That is, LLM evaluators rate each candidate system inconsistently and are dimension-dependent. They also struggle to compare candidates with close performance and become more unreliable with higher-quality summaries by obtaining a lower correlation with humans. In other words, with better abstractive summarization systems being introduced at a fast pace, LLMs may result in misleading and unreliable evaluations.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 37 of its 53 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

37 of 53 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
25 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  5. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  6. 2023Faithful Chain-of-Thought Reasoningcited by 285
  7. 2021Challenges in Detoxifying Language Modelscited by 169
  8. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  9. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  10. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  11. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  12. 2020Leveraging Graph to Improve Abstractive Multi-Document Summarizationcited by 67
  13. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  14. 2020UNION: An Unreferenced Metric for Evaluating Open-ended Story Generationcited by 60
  15. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  16. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  17. 2021MReD: A Meta-Review Dataset for Structure-Controllable Text Generationcited by 42
  18. 2018Improving Abstraction in Text Summarizationcited by 35
  19. 2017Deep Recurrent Generative Decoder for Abstractive Text Summarizationcited by 19
  20. 2018Abstractive Text Summarization by Incorporating Reader Commentscited by 15
  21. 2022Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Treescited by 14
  22. 2015Abstractive Multi-Document Summarization via Phrase Selection and Mergingcited by 10
  23. 2021Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attributioncited by 10
  24. 2022SentBS: Sentence-level Beam Search for Controllable Summarizationcited by 10
  25. 2021SgSum: Transforming Multi-document Summarization into Sub-graph Selectioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 36; 2025: 17; 2026: 11.

70 papers.

Sort:

By area:

  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  3. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  4. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  5. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  6. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  7. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  9. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  10. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  11. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 45
  12. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
38 more
  1. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  2. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  3. 2024PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Modelscited by 23
  4. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  5. 2024ExpertAF: Expert Actionable Feedback from Videocited by 17
  6. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluationcited by 16
  7. 2024METAL: Towards Multilingual Meta-Evaluationcited by 14
  8. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  9. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  10. 2024A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimizationcited by 11
  11. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  12. 2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10
  13. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 10
  14. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  15. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  16. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  17. 2025SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Paperscited by 8
  18. 2025Estimating LLM Consistency: A User Baseline vs Surrogate Metricscited by 8
  19. 2024Automated Review Generation Method Based on Large Language Modelscited by 7
  20. 2025Evaluating the Evaluators: Are readability metrics good measures of readability?cited by 6
  21. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimationcited by 6
  22. 2024Self-Improving Customer Review Response Generation Based on LLMscited by 5
  23. 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Modelscited by 5
  24. 2024FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devicescited by 5
  25. 2025KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Languagecited by 5
  26. 2025A Culturally-diverse Multilingual Multimodal Video Benchmark & Modelcited by 5
  27. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  28. 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluationscited by 5
  29. 2024Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMscited by 4
  30. 2024Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected valuescited by 4
  31. 2024HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settingscited by 4
  32. 2024Analyzing and Evaluating Correlation Measures in NLG Meta-Evaluationcited by 4
  33. 2023ODSum: New Benchmarks for Open Domain Multi-Document Summarizationcited by 3
  34. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  35. 2024EVA-Score: Evaluating Abstractive Long-form Summarization on Informativeness through Extraction and Validationcited by 3
  36. 2024Strategic Prompting for Conversational Tasks: A Comparative Analysis of Large Language Models Across Diverse Conversational Taskscited by 3
  37. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  38. 2025Towards EnergyGPT: A Large Language Model Specialized for the Energy Sectorcited by 3

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding LLM Reasoning for Abstractive Summarization
  2. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  3. 2023Large Language Models Are Not Strong Abstract Reasoners
  4. 2025DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?
  5. 2024A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks
  6. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
  7. 2023Evaluating Factual Consistency of Summaries with Large Language Models
  8. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization