mlsanityData to 14 Oct 2026

Less is More for Long Document Summary Evaluation by LLMs

arXiv 2309.07382 ↗First posted 14 Sep 2023cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown promising performance in summary evaluation tasks, yet they face challenges such as high computational costs and the Lost-in-the-Middle problem where important information in the middle of long documents is often overlooked. To address these issues, this paper introduces a novel approach, Extract-then-Evaluate, which involves extracting key sentences from a long source document and then evaluating the summary by prompting LLMs. The results reveal that the proposed method not only significantly reduces evaluation costs but also exhibits a higher correlation with human evaluations. Furthermore, we provide practical recommendations for optimal document length and sentence extraction methods, contributing to the development of cost-effective yet more accurate methods for LLM-based text generation evaluation.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

26 of 30 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
14 more
  1. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  2. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  7. 2023GPTScore: Evaluate as You Desirecited by 361
  8. 2021Efficient Attentions for Long Document Summarizationcited by 345
  9. 2020Evaluation of Text Generation: A Surveycited by 277
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  11. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  12. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  13. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  14. 2023A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 17; 2025: 7; 2026: 2.

28 papers.

Sort:

By area:

  1. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  2. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  3. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  4. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  5. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  6. 2023GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captionscited by 21
  7. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  8. 2025Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Educationcited by 14
  9. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12
  10. 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 11
  11. 2024Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Datacited by 11
  12. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
16 more
  1. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  2. 2025Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AIcited by 9
  3. 2024Do LLMs Agree on the Creativity Evaluation of Alternative Uses?cited by 6
  4. 2025LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructionscited by 6
  5. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  6. 2024Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metricscited by 5
  7. 2024Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Levelcited by 4
  8. 2025LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMscited by 3
  9. 2024LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reportscited by 2
  10. 2024SketchFill: Sketch-Guided Code Generation for Imputing Derived Missing Valuescited by 2
  11. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  12. 2025Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contextscited by 2
  13. 2024Generating Zero-shot Abstractive Explanations for Rumour Verificationcited by 1
  14. 2026Evaluating LLM-Based Grant Proposal Review via Structured Perturbationscited by 1
  15. 2024Harmful Suicide Content Detectioncited by 0
  16. 2026LOOMSUM: Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarizationcited by 0