# Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness arXiv: https://arxiv.org/abs/2205.01889 ยท HTML: https://mlsanity.com/paper/2205.01889/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 44 references that are in the map. - First posted on arXiv 4 May 2022. ## Builds on (most cited first, 29 of 29) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020) - [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019) - [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019) - [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019) - [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017) - [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016) - [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018) - [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019) - [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018) - [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020) - [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018) - [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018) - [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019) - [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://mlsanity.com/paper/2010.14235/) (2020) - [On Extractive and Abstractive Neural Document Summarization with Transformer Language Models](https://mlsanity.com/paper/1909.03186/) (2019) - [Leveraging Graph to Improve Abstractive Multi-Document Summarization](https://mlsanity.com/paper/2005.10043/) (2020) - [Learning to Extract Coherent Summary via Deep Reinforcement Learning](https://mlsanity.com/paper/1804.07036/) (2018) - [Learning Longer-term Dependencies in RNNs with Auxiliary Losses](https://mlsanity.com/paper/1803.00144/) (2018) - [Combinatorial Multi-Armed Bandit with General Reward Functions](https://mlsanity.com/paper/1610.06603/) (2016) - [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018) - [Noisy Self-Knowledge Distillation for Text Summarization](https://mlsanity.com/paper/2009.07032/) (2020) - [Learning Opinion Summarizers by Selecting Informative Reviews](https://mlsanity.com/paper/2109.04325/) (2021) - [Generating Summaries with Topic Templates and Structured Convolutional Decoders](https://mlsanity.com/paper/1906.04687/) (2019) - [AgreeSum: Agreement-Oriented Multi-Document Summarization](https://mlsanity.com/paper/2106.02278/) (2021) - [TWAG: A Topic-Guided Wikipedia Abstract Generator](https://mlsanity.com/paper/2106.15135/) (2021) ## Built on by (most cited first, 13 of 13) - [Conditional Generation with a Question-Answering Blueprint](https://mlsanity.com/paper/2207.00397/) (2022) - [Do Multi-Document Summarization Models Synthesize?](https://mlsanity.com/paper/2301.13844/) (2023) - [Generating EDU Extracts for Plan-Guided Summary Re-Ranking](https://mlsanity.com/paper/2305.17779/) (2023) - [SINC: Self-Supervised In-Context Learning for Vision-Language Tasks](https://mlsanity.com/paper/2307.07742/) (2023) - [A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2305.08503/) (2023) - [A Mixed-Language Multi-Document News Summarization Dataset and a Graphs-Based Extract-Generate Model](https://mlsanity.com/paper/2410.09773/) (2024) - [ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference](https://mlsanity.com/paper/2509.03565/) (2025) - [SPEC: Summary Preference Decomposition for Low-Resource Abstractive Summarization](https://mlsanity.com/paper/2303.14011/) (2023) - [SKT5SciSumm -- Revisiting Extractive-Generative Approach for Multi-Document Scientific Summarization](https://mlsanity.com/paper/2402.17311/) (2024) - [Disentangling Specificity for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2406.00005/) (2024) - [Principled Content Selection to Generate Diverse and Personalized Multi-Document Summaries](https://mlsanity.com/paper/2505.21859/) (2025) - [Extract-and-Abstract: Unifying Extractive and Abstractive Summarization within Single Encoder-Decoder Framework](https://mlsanity.com/paper/2409.11827/) (2024) - [CCSK:Cognitive Convection of Self-Knowledge Based Retrieval Augmentation for Large Language Models](https://mlsanity.com/paper/2504.10498/) (2025) ## Abstract (arXiv) A notable challenge in Multi-Document Summarization (MDS) is the extremely-long length of the input. In this paper, we present an extract-then-abstract Transformer framework to overcome the problem. Specifically, we leverage pre-trained language models to construct a hierarchical extractor for salient sentence selection across documents and an abstractor for rewriting the selected contents as summaries. However, learning such a framework is challenging since the optimal contents for the abstractor are generally unknown. Previous works typically create pseudo extraction oracle to enable the supervised learning for both the extractor and the abstractor. Nevertheless, we argue that the performance of such methods could be restricted due to the insufficient information for prediction and inconsistent objectives between training and testing. To this end, we propose a loss weighting mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle, and leverage the fine-tuned abstractor to generate summary references as auxiliary signals for learning the extractor. Moreover, we propose a reinforcement learning method that can efficiently apply to the extractor for harmonizing the optimization between training and testing. Experiment results show that our framework substantially outperforms strong baselines with comparable model sizes and achieves the best results on the Multi-News, Multi-XScience, and WikiCatSum corpora.