mlsanityData to 7 Oct 2026

Human-like Summarization Evaluation with ChatGPT

arXiv 2304.02554 ↗First posted 5 Apr 2023cs.CLcited by 126

Not on the map. Why?

Abstract (arXiv)

Evaluating text summarization is a challenging problem, and existing evaluation metrics are far from satisfactory. In this study, we explored ChatGPT's ability to perform human-like summarization evaluation using four human evaluation methods on five datasets. We found that ChatGPT was able to complete annotations relatively smoothly using Likert scale scoring, pairwise comparison, Pyramid, and binary factuality evaluation. Additionally, it outperformed commonly used automatic evaluation metrics on some datasets. Furthermore, we discussed the impact of different prompts, compared its performance with that of human evaluation, and analyzed the generated explanations and invalid responses.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

17 of 22 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  11. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
5 more
  1. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  2. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  3. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  4. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  5. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 39; 2024: 43; 2025: 36; 2026: 8.

126 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  4. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  6. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  7. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  8. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  9. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  10. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  11. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  12. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
38 more
  1. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  4. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  5. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  8. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  9. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  10. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  11. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  12. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  13. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  14. 2023Summarization is (Almost) Deadcited by 43
  15. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  16. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  17. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  18. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  19. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 35
  20. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 35
  21. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  22. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  23. 2023Interactive Natural Language Processingcited by 31
  24. 2023Aligning Large Language Models with Human: A Surveycited by 31
  25. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  26. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  27. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 29
  28. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  29. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
  30. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  31. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  32. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 23
  33. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23
  34. 2023Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Modelcited by 21
  35. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  36. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 19
  37. 2023Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directionscited by 18
  38. 2025LAMD: Context-driven Android Malware Detection and Classification with LLMscited by 18

50 most cited of 126 All 126 (JSON)