mlsanityData to 14 Oct 2026

SQuALITY: Building a Long-Document Summarization Dataset the Hard Way

arXiv 2205.11465 ↗First posted 23 May 2022cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries -- which are nearly always in difficult-to-work-with technical domains -- or by using approximate heuristics to extract them from everyday text -- which frequently yields unfaithful summaries. In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualified contractors to read stories and write original summaries from scratch. To amortize reading time, we collect five summaries per document, with the first giving an overview and the subsequent four addressing specific questions. We use this protocol to collect SQuALITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

35 of 49 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  10. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  11. 2021Efficient Attentions for Long Document Summarizationcited by 345
  12. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
23 more
  1. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  2. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  4. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  5. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  6. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  7. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  8. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  9. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 156
  10. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  11. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  12. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  13. 2020WikiAsp: A Dataset for Multi-domain Aspect-based Summarizationcited by 56
  14. 2021Exploring Neural Models for Query-Focused Summarizationcited by 44
  15. 2017Diversity driven Attention Model for Query-based Abstractive Summarizationcited by 42
  16. 2020AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarizationcited by 42
  17. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  18. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  19. 2020Exploring Content Selection in Summarization of Novel Chapterscited by 29
  20. 2018Efficient Online Scalar Annotation with Bounded Supportcited by 27
  21. 2022HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarizationcited by 23
  22. 2021How well do you know your summarization datasets?cited by 16
  23. 2021AgreeSum: Agreement-Oriented Multi-Document Summarizationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 19; 2025: 10; 2026: 2.

50 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  4. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  5. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  6. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  7. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  8. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  9. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  10. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  11. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  12. 2025PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Pathscited by 61
38 more
  1. 2024Synthetic continued pretrainingcited by 60
  2. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
  3. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  4. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  5. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  6. 2022OASum: Large-Scale Open Domain Aspect-based Summarizationcited by 24
  7. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  8. 2024MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflowscited by 19
  9. 2022Socratic Pretraining: Question-Driven Pretraining for Controllable Summarizationcited by 17
  10. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  11. 2023LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigmscited by 14
  12. 2024Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphscited by 12
  13. 2023Using Natural Language Explanations to Rescale Human Judgmentscited by 11
  14. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  15. 2024LexAbSumm: Aspect-based Summarization of Legal Decisionscited by 9
  16. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  17. 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 8
  18. 2024Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problemscited by 7
  19. 2026Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guaranteescited by 7
  20. 2023The Current State of Summarizationcited by 6
  21. 2023Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Studycited by 5
  22. 2024Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Textscited by 5
  23. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  24. 2022What is Wrong with Language Models that Can Not Tell a Story?cited by 4
  25. 2023QontSum: On Contrasting Salient Content for Query-focused Summarizationcited by 4
  26. 2025AcademicEval: Live Long-Context LLM Benchmarkcited by 4
  27. 2023ODSum: New Benchmarks for Open Domain Multi-Document Summarizationcited by 3
  28. 2024MODABS: Multi-Objective Learning for Dynamic Aspect-Based Summarizationcited by 3
  29. 2024Learning to Rank Salient Content for Query-focused Summarizationcited by 3
  30. 2024The State and Fate of Summarization Datasets: A Surveycited by 3
  31. 2025Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimizationcited by 3
  32. 2024IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarizationcited by 2
  33. 2024Revealing Trends in Datasets from the 2022 ACL and EMNLP Conferencescited by 0
  34. 2024Does This Summary Answer My Question? Modeling Query-Focused Summary Readers with Rational Speech Actscited by 0
  35. 2025Advancing AI Research Assistants with Expert-Involved Learningcited by 0
  36. 2025MSRS: Evaluating Multi-Source Retrieval-Augmented Generationcited by 0
  37. 2025LC-Eval: A Bilingual Multi-Task Evaluation Benchmark for Long-Context Understandingcited by 0
  38. 2026Query-Focused Event Summarization: A Dataset and Benchmarkcited by 0