# Summarization is (Almost) Dead

arXiv: https://arxiv.org/abs/2309.09558 · HTML: https://mlsanity.com/paper/2309.09558/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method](https://mlsanity.com/paper/2305.13412/) (2023)
- [NCLS: Neural Cross-Lingual Summarization](https://mlsanity.com/paper/1909.00156/) (2019)
- [ClidSum: A Benchmark Dataset for Cross-Lingual Dialogue Summarization](https://mlsanity.com/paper/2202.05599/) (2022)
- [PyTorrent: A Python Library Corpus for Large-scale Language Models](https://mlsanity.com/paper/2110.01710/) (2021)
- [Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks](https://mlsanity.com/paper/2305.15044/) (2023)
- [Unsupervised Multi-Granularity Summarization](https://mlsanity.com/paper/2201.12502/) (2022)

## Built on by (most cited first, 30 of 44)

- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments](https://mlsanity.com/paper/2501.10893/) (2025)
- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks](https://mlsanity.com/paper/2405.20099/) (2024)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [Learning to Break: Knowledge-Enhanced Reasoning in Multi-Agent Debate System](https://mlsanity.com/paper/2312.04854/) (2023)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [Preemptive Answer "Attacks" on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2405.20902/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding](https://mlsanity.com/paper/2405.19567/) (2024)
- [Translating Legalese: Enhancing Public Understanding of Court Opinions with Legal Summarizers](https://mlsanity.com/paper/2311.06534/) (2023)
- [Key-Element-Informed sLLM Tuning for Document Summarization](https://mlsanity.com/paper/2406.04625/) (2024)
- [Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoning](https://mlsanity.com/paper/2501.07845/) (2025)
- [Automated Review Generation Method Based on Large Language Models](https://mlsanity.com/paper/2407.20906/) (2024)
- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment](https://mlsanity.com/paper/2409.16145/) (2024)
- [Opportunities and Challenges of Generative-AI in Finance](https://mlsanity.com/paper/2410.15653/) (2024)
- [MageBench: Bridging Large Multimodal Models to Agents](https://mlsanity.com/paper/2412.04531/) (2024)
- [Characterizing Multimodal Long-form Summarization: A Case Study on Financial Reports](https://mlsanity.com/paper/2404.06162/) (2024)
- [A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs](https://mlsanity.com/paper/2502.09921/) (2025)
- [From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper](https://mlsanity.com/paper/2503.07450/) (2025)
- [Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarization](https://mlsanity.com/paper/2509.00529/) (2025)
- [Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization](https://mlsanity.com/paper/2510.19325/) (2025)
- [Rethinking Hate Speech Detection on Social Media: Can LLMs Replace Traditional Models?](https://mlsanity.com/paper/2506.12744/) (2025)
- [Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications](https://mlsanity.com/paper/2601.08833/) (2025)
- [From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning](https://mlsanity.com/paper/2602.23729/) (2026)
- [AI-Enhanced Cognitive Behavioral Therapy: Deep Learning and Large Language Models for Extracting Cognitive Pathways from Social Media Texts](https://mlsanity.com/paper/2404.11449/) (2024)

## Abstract (arXiv)

How well can large language models (LLMs) generate summaries? We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of LLMs across five distinct summarization tasks. Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models. Specifically, LLM-generated summaries exhibit better factual consistency and fewer instances of extrinsic hallucinations. Due to the satisfactory performance of LLMs in summarization tasks (even surpassing the benchmark of reference summaries), we believe that most conventional works in the field of text summarization are no longer necessary in the era of LLMs. However, we recognize that there are still some directions worth exploring, such as the creation of novel datasets with higher quality and more reliable evaluation methods.