# Less is More for Long Document Summary Evaluation by LLMs

arXiv: https://arxiv.org/abs/2309.07382 · HTML: https://mlsanity.com/paper/2309.07382/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://mlsanity.com/paper/2301.13298/) (2023)
- [On the Limitations of Reference-Free Evaluations of Generated Text](https://mlsanity.com/paper/2210.12563/) (2022)
- [How Far are We from Robust Long Abstractive Summarization?](https://mlsanity.com/paper/2210.16732/) (2022)
- [A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization](https://mlsanity.com/paper/2303.03948/) (2023)

## Built on by (most cited first, 28 of 28)

- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [NExtLong: Toward Effective Long-Context Training without Long Documents](https://mlsanity.com/paper/2501.12766/) (2025)
- [Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Education](https://mlsanity.com/paper/2503.19182/) (2025)
- [ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage](https://mlsanity.com/paper/2410.16848/) (2024)
- [Attribute or Abstain: Large Language Models as Long Document Assistants](https://mlsanity.com/paper/2407.07799/) (2024)
- [Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data](https://mlsanity.com/paper/2410.11996/) (2024)
- [AmbigNLG: Addressing Task Ambiguity in Instruction for NLG](https://mlsanity.com/paper/2402.17717/) (2024)
- [Characterizing Large Language Models as Rationalizers of Knowledge-intensive Tasks](https://mlsanity.com/paper/2311.05085/) (2023)
- [Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI](https://mlsanity.com/paper/2504.08148/) (2025)
- [Do LLMs Agree on the Creativity Evaluation of Alternative Uses?](https://mlsanity.com/paper/2411.15560/) (2024)
- [LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions](https://mlsanity.com/paper/2505.17134/) (2025)
- [Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy](https://mlsanity.com/paper/2402.12821/) (2024)
- [Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics](https://mlsanity.com/paper/2410.10867/) (2024)
- [Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Level](https://mlsanity.com/paper/2405.10202/) (2024)
- [LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs](https://mlsanity.com/paper/2509.15568/) (2025)
- [LaMSUM: Amplifying Voices Against Harassment through LLM Guided Extractive Summarization of User Incident Reports](https://mlsanity.com/paper/2406.15809/) (2024)
- [SketchFill: Sketch-Guided Code Generation for Imputing Derived Missing Values](https://mlsanity.com/paper/2412.19113/) (2024)
- [Reranking-based Generation for Unbiased Perspective Summarization](https://mlsanity.com/paper/2506.15925/) (2025)
- [Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts](https://mlsanity.com/paper/2508.19578/) (2025)
- [Generating Zero-shot Abstractive Explanations for Rumour Verification](https://mlsanity.com/paper/2401.12713/) (2024)
- [Evaluating LLM-Based Grant Proposal Review via Structured Perturbations](https://mlsanity.com/paper/2603.08281/) (2026)
- [Harmful Suicide Content Detection](https://mlsanity.com/paper/2407.13942/) (2024)
- [LOOMSUM: Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarization](https://mlsanity.com/paper/2609.00241/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promising performance in summary evaluation tasks, yet they face challenges such as high computational costs and the Lost-in-the-Middle problem where important information in the middle of long documents is often overlooked. To address these issues, this paper introduces a novel approach, Extract-then-Evaluate, which involves extracting key sentences from a long source document and then evaluating the summary by prompting LLMs. The results reveal that the proposed method not only significantly reduces evaluation costs but also exhibits a higher correlation with human evaluations. Furthermore, we provide practical recommendations for optimal document length and sentence extraction methods, contributing to the development of cost-effective yet more accurate methods for LLM-based text generation evaluation.