mlsanityData to 7 Oct 2026

Benchmarking Large Language Models for News Summarization

arXiv 2301.13848 ↗First posted 31 Jan 2023cs.CL cs.AI cs.LGcited by 444

Not on the map. Why?

Cited by 27 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. By conducting a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales, we make two important observations. First, we find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability. Second, existing studies have been limited by low-quality references, leading to underestimates of human performance and lower few-shot and finetuning performance. To better evaluate LLMs, we perform human evaluation over high-quality summaries we collect from freelance writers. Despite major stylistic differences such as the amount of paraphrasing, we find that LMM summaries are judged to be on par with human written summaries.

Key facts. Cited by 444 papers in the map (data through 7 Oct 2026). Builds on 40 of its 76 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

40 of 76 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
28 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2015Teaching Machines to Read and Comprehendcited by 1,039
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  13. 2019Text Summarization with Pretrained Encoderscited by 553
  14. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  15. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  16. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  17. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  18. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  19. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  20. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  21. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  22. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  23. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  24. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  25. 2020BLEU might be Guilty but References are not Innocentcited by 109
  26. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  27. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  28. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 91; 2024: 180; 2025: 120; 2026: 53.

444 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  8. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  9. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  10. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  11. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  12. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
38 more
  1. 2024Tool Learning with Large Language Models: A Surveycited by 221
  2. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  3. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  4. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  5. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  6. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  7. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  8. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  9. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  10. 2024Large Language Models Meet NLP: A Surveycited by 102
  11. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  12. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  13. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  14. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  15. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  16. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  17. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  18. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  19. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  20. 2023Human Feedback is not Gold Standardcited by 82
  21. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  22. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  23. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  24. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  25. 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 75
  26. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  27. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  28. 2024MarsCode Agent: AI-native Automated Bug Fixingcited by 68
  29. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  30. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  31. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  32. 2023Graph Neural Prompting with Large Language Modelscited by 64
  33. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  34. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  35. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 60
  36. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  37. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  38. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58

50 most cited of 444 All 444 (JSON)