# Multi-Document Summarization with Centroid-Based Pretraining

arXiv: https://arxiv.org/abs/2208.01006 · HTML: https://mlsanity.com/paper/2208.01006/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 1 Aug 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal](https://mlsanity.com/paper/2005.10070/) (2020)
- [PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization](https://mlsanity.com/paper/2110.08499/) (2021)
- [Generating Representative Headlines for News Stories](https://mlsanity.com/paper/2001.09386/) (2020)
- [An Unsupervised Masking Objective for Abstractive Multi-Document News Summarization](https://mlsanity.com/paper/2201.02321/) (2022)

## Built on by (most cited first, 5 of 5)

- [A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2305.08503/) (2023)
- [PELMS: Pre-training for Effective Low-Shot Multi-Document Summarization](https://mlsanity.com/paper/2311.09836/) (2023)
- [A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization](https://mlsanity.com/paper/2504.16711/) (2025)
- [BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion](https://mlsanity.com/paper/2603.11415/) (2026)
- [A Training-Free Mixture-of-Agents Framework for Multi-Document Summarization using LLMs and Knowledge Graphs](https://mlsanity.com/paper/2606.03867/) (2026)

## Abstract (arXiv)

In Multi-Document Summarization (MDS), the input can be modeled as a set of documents, and the output is its summary. In this paper, we focus on pretraining objectives for MDS. Specifically, we introduce a novel pretraining objective, which involves selecting the ROUGE-based centroid of each document cluster as a proxy for its summary. Our objective thus does not require human written summaries and can be utilized for pretraining on a dataset consisting solely of document sets. Through zero-shot, few-shot, and fully supervised experiments on multiple MDS datasets, we show that our model Centrum is better or comparable to a state-of-the-art model. We make the pretrained and fine-tuned models freely available to the research community https://github.com/ratishsp/centrum.