mlsanityData to 14 Oct 2026

Benchmarking LLMs on the Semantic Overlap Summarization Task

arXiv 2402.17008 ↗First posted 26 Feb 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Semantic Overlap Summarization (SOS) is a constrained multi-document summarization task, where the constraint is to capture the common/overlapping information between two alternative narratives. In this work, we perform a benchmarking study of popular Large Language Models (LLMs) exclusively on the SOS task. Additionally, we introduce the PrivacyPolicyPairs (3P) dataset to expand the space of SOS benchmarks in terms of quantity and variety. This dataset provides 135 high-quality SOS data samples sourced from privacy policy documents. We then use a standard prompting taxonomy called TELeR to create and evaluate 905,216 distinct LLM-generated summaries over two SOS datasets from different domains, and we further conduct human evaluation on a subset of 540 samples. We conclude the paper by analyzing models' performances and the reliability of automatic evaluation. The code and datasets used to conduct this study are available at https://anonymous.4open.science/r/llm_eval-E16D.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 54 of its 101 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

54 of 101 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  9. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  11. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  12. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  13. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  14. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  15. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  16. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  17. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 222
  18. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  19. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  20. 2020Extractive Summarization as Text Matchingcited by 189
  21. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  22. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  23. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  24. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  25. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  26. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  27. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  28. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  29. 2020Multi-document Summarization via Deep Learning Techniques: A Surveycited by 58
  30. 2017Graph-based Neural Multi-Document Summarizationcited by 54
  31. 2023TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Taskscited by 49
  32. 2018A Unified Model for Extractive and Abstractive Summarization using Inconsistency Losscited by 47
  33. 2023Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articlescited by 45
  34. 2018Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarizationcited by 42
  35. 2020ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generationcited by 40
  36. 2019Searching for Effective Neural Extractive Summarization: What Works and What's Nextcited by 38
  37. 2020SummPip: Unsupervised Multi-Document Summarization with Sentence Graph Compressioncited by 24
  38. 2017Generative Adversarial Network for Abstractive Text Summarizationcited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025LLM for Complex Reasoning Task: An Exploratory Study in Fermi Problemscited by 1
  2. 2025LLM for Comparative Narrative Analysiscited by 1