mlsanityData to 7 Oct 2026

FineSurE: Fine-grained Summarization Evaluation using LLMs

arXiv 2407.00908 ↗First posted 1 Jul 2024cs.CL cs.AIcited by 83

Not on the map. Why?

Abstract (arXiv)

Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and time-consuming nature of human evaluation. Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likert-scale scores. This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations. To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs). It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment. We compare various open-source and proprietary LLMs as backbones for FineSurE. In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions. The code is available at https://github.com/DISL-Lab/FineSurE-ACL24.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

34 of 49 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
22 more
  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  2. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  5. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  6. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  10. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  11. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  12. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  13. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  14. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  15. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  16. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  17. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  18. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  19. 2022Revisiting text decomposition methods for NLI-based factuality scoring of summariescited by 14
  20. 2023OpinSummEval: Revisiting Automated Evaluation for Opinion Summarizationcited by 13
  21. 2023Enhancing Abstractiveness of Summarization Models through Calibrated Distillationcited by 8
  22. 2023SETI: Systematicity Evaluation of Textual Inferencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 44; 2026: 24.

83 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025LongCodeZip: Compress Long Context for Code Language Modelscited by 38
  4. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?cited by 37
  5. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  7. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  8. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 22
  9. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  10. 2025AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMscited by 12
  11. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  12. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
38 more
  1. 2025Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Modelscited by 10
  2. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubriccited by 8
  3. 2025StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMscited by 7
  4. 2026EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generationcited by 7
  5. 2024Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summariescited by 6
  6. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  7. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarizationcited by 6
  8. 2025Evaluating the Evaluators: Are readability metrics good measures of readability?cited by 6
  9. 2024Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Modelscited by 5
  10. 2024CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinionscited by 5
  11. 2025Summarizing Speech: A Comprehensive Surveycited by 5
  12. 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captionscited by 5
  13. 2025LM Agents for Coordinating Multi-User Information Gatheringcited by 4
  14. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4
  15. 2025Long-Form Information Alignment Evaluation Beyond Atomic Factscited by 4
  16. 2025SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modellingcited by 4
  17. 2026Learning to Control Summaries with Score Rankingcited by 4
  18. 2024EVA-Score: Evaluating Abstractive Long-form Summarization on Informativeness through Extraction and Validationcited by 3
  19. 2024Disentangling Likes and Dislikes in Personalized Generative Explainable Recommendationcited by 3
  20. 2024Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarizationcited by 3
  21. 2024VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Modelscited by 3
  22. 2025Large Language Models for Knowledge Graph Embedding: A Surveycited by 3
  23. 2025A Rate-Distortion Framework for Summarizationcited by 3
  24. 2025DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Managementcited by 3
  25. 2025Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasingcited by 3
  26. 2025Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guaranteescited by 3
  27. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  28. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  29. 2026Calibrating Model-Based Evaluation Metrics for Summarizationcited by 3
  30. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  31. 2026Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Whycited by 3
  32. 2024CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novelscited by 2
  33. 2024Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Modelscited by 2
  34. 2024Chain-of-MetaWriting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Textscited by 2
  35. 2025Rehearse With User: Personalized Opinion Summarization via Role-Playing based on Large Language Modelscited by 2
  36. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  37. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  38. 2025QUARTZ : QA-based Unsupervised Abstractive Refinement for Task-oriented Dialogue Summarizationcited by 2

50 most cited of 83 All 83 (JSON)