mlsanityData to 7 Oct 2026

News Summarization and Evaluation in the Era of GPT-3

arXiv 2209.12356 ↗First posted 26 Sep 2022cs.CLcited by 324

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent success of prompting large language models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Finally, we evaluate models on a setting beyond generic summarization, specifically keyword-based summarization, and show how dominant fine-tuning approaches compare to prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and prompt-based models across 4 standard summarization benchmarks, (b) 1K human preference judgments comparing different systems for generic- and keyword-based summarization.

Key facts. Cited by 324 papers in the map (data through 7 Oct 2026). Builds on 54 of its 85 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

54 of 85 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
38 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  6. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  12. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  13. 2022Self-critiquing models for assisting human evaluatorscited by 372
  14. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  15. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  16. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  17. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  18. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  19. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  20. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  21. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  22. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  23. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  24. 2020With Little Power Comes Great Responsibilitycited by 138
  25. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  26. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  27. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  28. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  29. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  30. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 87
  31. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  32. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  33. 2021ASPECTNEWS: Aspect-Oriented Summarization of News Documentscited by 37
  34. 2022Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metricscited by 37
  35. 2022Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarizationcited by 29
  36. 2021An Exploratory Study on Long Dialogue Summarization: What Works and What's Nextcited by 28
  37. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  38. 2021Training Dynamics for Text Summarization Modelscited by 25

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 120; 2024: 119; 2025: 51; 2026: 14.

324 papers.

Sort:

By area:

  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  5. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  9. 2023Are aligned neural networks adversarially aligned?cited by 305
  10. 2023How is ChatGPT's behavior changing over time?cited by 265
  11. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  12. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
38 more
  1. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  3. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  4. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  5. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  6. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  7. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  8. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  9. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  10. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  11. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  12. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  13. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  14. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  15. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  16. 2024Large Language Models Meet NLP: A Surveycited by 102
  17. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  18. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  19. 2023Shepherd: A Critic for Language Model Generationcited by 96
  20. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  21. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  22. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  23. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  24. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  25. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  26. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  27. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  28. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  29. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  30. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  31. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  32. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  33. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58
  34. 2024Large Language Models Reflect the Ideology of their Creatorscited by 56
  35. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  36. 2022LENS: A Learnable Evaluation Metric for Text Simplificationcited by 54
  37. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  38. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53

50 most cited of 324 All 324 (JSON)