mlsanityData to 7 Oct 2026

Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing

arXiv 2406.03993 ↗First posted 6 Jun 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles. However, little is known about the robustness of such a process of zero-shot summarization. To bridge this gap, we propose relevance paraphrasing, a simple strategy that can be used to measure the robustness of LLMs as summarizers. The relevance paraphrasing approach identifies the most relevant sentences that contribute to generating an ideal summary, and then paraphrases these inputs to obtain a minimally perturbed dataset. Then, by evaluating model performance for summarization on both the original and perturbed datasets, we can assess the LLM's one aspect of robustness. We conduct extensive experiments with relevance paraphrasing on 4 diverse datasets, as well as 4 LLMs of different sizes (GPT-3.5-Turbo, Llama-2-13B, Mistral-7B, and Dolly-v2-7B). Our results indicate that LLMs are not consistent summarizers for the minimally perturbed articles, necessitating further improvements.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 27 of its 52 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

27 of 52 references.

References · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  11. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  12. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
15 more
  1. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  2. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  3. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  4. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  5. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  6. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 63
  7. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30
  8. 2023Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibilitycited by 28
  9. 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
  10. 2022Towards Improving Faithfulness in Abstractive Summarizationcited by 27
  11. 2020CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systemscited by 17
  12. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  13. 2023Robust Natural Language Understanding with Residual Attention Debiasingcited by 10
  14. 2018Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Informationcited by 8
  15. 2022Improving the Robustness of Summarization Models by Detecting and Removing Input Noisecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  2. 2025Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarizationcited by 6
  3. 2025Re-ranking Using Large Language Models for Mitigating Exposure to Harmful Content on Social Media Platformscited by 2
  4. 2025"Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selectioncited by 1
  5. 2025Evaluating book summaries from internal knowledge in Large Language Models: a cross-model and semantic consistency approachcited by 1
  6. 2025Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languagescited by 1
  7. 2025Enhancing Long Document Long Form Summarisation with Self-Planningcited by 1
  8. 2025The Effect of Document Summarization on LLM-Based Relevance Judgmentscited by 0
  9. 2026LLM or Human? Perceptions of Trust and Information Quality in Research Summariescited by 0
  10. 2026A Systematic Evaluation of Large Language Models for PTSD Severity Estimation: The Role of Contextual Knowledge and Modeling Strategiescited by 0
  11. 2026Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?cited by 0