mlsanityData to 13 Oct 2026

BooookScore: A systematic exploration of book-length summarization in the era of LLMs

arXiv 2310.00785 ↗First posted 1 Oct 2023cs.CL cs.AI cs.LGcited by 137

Not on the map. Why?

Abstract (arXiv)

Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving 15K USD and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than those generated by open-source models. While LLaMA 2 falls behind other models, Mixtral achieves performance on par with GPT-3.5-Turbo. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by annotators.

Key facts. Cited by 137 papers in the map (data through 13 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

34 of 41 references.

References · 34

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,412
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 668
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  9. 2023GPTScore: Evaluate as You Desirecited by 361
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 334
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 326
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 290
22 more
  1. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  2. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  3. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  4. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  5. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  6. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  7. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  8. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  9. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  10. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  11. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  12. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  13. 2023Summarization is (Almost) Deadcited by 44
  14. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  15. 2020Inquisitive Question Generation for High Level Text Comprehensioncited by 31
  16. 2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
  17. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  18. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  19. 2023A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documentscited by 15
  20. 2023Elaborative Simplification as Implicit Questions Under Discussioncited by 14
  21. 2023FOLLOWUPQG: Towards Information-Seeking Follow-up Question Generationcited by 12
  22. 2023AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Contentcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 56; 2025: 59; 2026: 20.

137 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 813
  2. 2024How to Train Long-Context Language Models (Effectively)cited by 144
  3. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 124
  4. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 122
  5. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 120
  6. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 117
  7. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  8. 2025Recursive Language Modelscited by 99
  9. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  10. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 86
  11. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 72
  12. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 72
38 more
  1. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  2. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  3. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 51
  4. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  5. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  6. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  7. 2024Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Workscited by 38
  8. 2024Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?cited by 34
  9. 2024Semantic Operators: A Declarative Model for Rich, AI-based Data Processingcited by 31
  10. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  11. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
  12. 2024Embedding-based classifiers can detect prompt injection attackscited by 27
  13. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  14. 2024On the Evaluation of Machine-Generated Reportscited by 23
  15. 2024Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarizationcited by 23
  16. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  17. 2025YourBench: Easy Custom Evaluation Sets for Everyonecited by 20
  18. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  19. 2024On Large Language Models in National Security Applicationscited by 18
  20. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 18
  21. 2023Fair Abstractive Summarization of Diverse Perspectivescited by 17
  22. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  23. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  24. 2024A Modular Approach for Multimodal Summarization of TV Showscited by 13
  25. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  26. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  27. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 13
  28. 2024ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coveragecited by 12
  29. 2025LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitationscited by 12
  30. 2025HICode: Hierarchical Inductive Coding with LLMscited by 12
  31. 2025HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insightscited by 11
  32. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 10
  33. 2025Does quantization affect models' performance on long-context tasks?cited by 10
  34. 2025NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarizationcited by 10
  35. 2026HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playingcited by 10
  36. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9
  37. 2024Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarizationcited by 9
  38. 2025Agent-as-Judge for Factual Summarization of Long Narrativescited by 9

50 most cited of 137 All 137 (JSON)