mlsanityData to 7 Oct 2026

Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles

arXiv 2309.09369 ↗First posted 17 Sep 2023cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Previous research in multi-document news summarization has typically concentrated on collating information that all sources agree upon. However, the summarization of diverse information dispersed across multiple articles about an event remains underexplored. In this paper, we propose a new task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, we outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. The dataset includes 245 news stories, with each story comprising 10 news articles and paired with a human-validated reference. Next, to enable consistent automatic evaluation, we conducted a comprehensive analysis to pinpoint the position and verbosity biases when utilizing Large Language Model (LLM)-based metrics for evaluating the coverage and faithfulness of summaries. Through correlation analyses, we outline the best practices for effectively using automatic LLM-based metrics on the DiverseSumm dataset. Finally, we study how LLMs summarize multiple news articles by analyzing which type of diverse information LLMs are capable of identifying. Our analyses suggest that despite the extraordinary capabilities of LLMs in single-document summarization, the proposed task remains a complex challenge for them mainly due to their limited coverage, with GPT-4 only able to cover under 40% of the diverse information on average.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 17 Sep 2023.

Builds on

26 of 50 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  9. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  10. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  11. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  12. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
14 more
  1. 2016Neural Summarization by Extracting Sentences and Wordscited by 114
  2. 2020Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articlescited by 96
  3. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  4. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  5. 2020A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portalcited by 72
  6. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  7. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  8. 2023Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluationcited by 14
  9. 2019Guiding Extractive Summarization with Question-Answering Rewardscited by 13
  10. 2016Multi-document abstractive summarization using ILP based multi-sentence compressioncited by 11
  11. 2022Discord Questions: A Computational Approach To Diversity Analysis in News Coveragecited by 11
  12. 2023SWING: Balancing Coverage and Faithfulness for Dialogue Summarizationcited by 10
  13. 2018Towards a Neural Network Approach to Abstractive Multi-Document Summarizationcited by 6
  14. 2023Designing and Evaluating Interfaces that Highlight News Coverage Diversity Using Discord Questionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 17; 2025: 17; 2026: 6.

45 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  3. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  4. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  5. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  6. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  7. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 69
  8. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  9. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  10. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48
  11. 2025CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactionscited by 46
  12. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
33 more
  1. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  2. 2023Fair Abstractive Summarization of Diverse Perspectivescited by 17
  3. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  4. 2024VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Modelscited by 16
  5. 2025WikiVideo: Article Generation from Multiple Videoscited by 13
  6. 2024Open-World Evaluation for Retrieving Diverse Perspectivescited by 11
  7. 2025Behavioral Analysis of Information Salience in Large Language Modelscited by 11
  8. 2025Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMscited by 11
  9. 2024SafeWorld: Geo-Diverse Safety Alignmentcited by 10
  10. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaborationcited by 9
  11. 2024Which questions should I answer? Salience Prediction of Inquisitive Questionscited by 8
  12. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  13. 2024Verbosity neq Veracity: Demystify Verbosity Compensation Behavior of Large Language Modelscited by 5
  14. 2025MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collectionscited by 4
  15. 2025GenerationPrograms: Fine-grained Attribution with Executable Programscited by 4
  16. 2025Uncovering the Bigger Picture: Comprehensive Event Understanding Via Diverse News Retrievalcited by 4
  17. 2024StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputscited by 3
  18. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approachescited by 3
  19. 2025VeriTrail: Closed-Domain Hallucination Detection with Traceabilitycited by 3
  20. 2024Benchmarking LLMs on the Semantic Overlap Summarization Taskcited by 2
  21. 2025ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inferencecited by 2
  22. 2026Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarizationcited by 2
  23. 2024QANA: LLM-based Question Generation and Network Analysis for Zero-shot Key Point Analysis and Beyondcited by 1
  24. 2025Principled Content Selection to Generate Diverse and Personalized Multi-Document Summariescited by 1
  25. 2025Enhancing Long Document Long Form Summarisation with Self-Planningcited by 1
  26. 2026When Bigger Isn't Better: A Comprehensive Fairness Evaluation of Political Bias in Multi-News Summarisationcited by 1
  27. 2024Cross-Document Event-Keyed Summarizationcited by 0
  28. 2025Improving Fairness of Large Language Models in Multi-document Summarizationcited by 0
  29. 2025Comparative Personalization for Multi-document Summarizationcited by 0
  30. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  31. 2026Summarization is Not Dead Yetcited by 0
  32. 2026Decoupling Generation and Selection for Budget-Constrained Faithful Summarizationcited by 0
  33. 2026Do we need to answer that question? Salience and Answerability of Potential Questions in Naturalistic Dialoguecited by 0