# Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents

arXiv: https://arxiv.org/abs/2205.12486 · HTML: https://mlsanity.com/paper/2205.12486/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [GSum: A General Framework for Guided Neural Abstractive Summarization](https://mlsanity.com/paper/2010.08014/) (2020)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [CTRLsum: Towards Generic Controllable Text Summarization](https://mlsanity.com/paper/2012.04281/) (2020)
- [Controlling Output Length in Neural Encoder-Decoders](https://mlsanity.com/paper/1609.09552/) (2016)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [Positional Encoding to Control Output Sequence Length](https://mlsanity.com/paper/1904.07418/) (2019)
- [DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization](https://mlsanity.com/paper/2110.08168/) (2021)
- [HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization](https://mlsanity.com/paper/2203.10741/) (2022)
- [Convex Aggregation for Opinion Summarization](https://mlsanity.com/paper/2104.01371/) (2021)
- [Controllable Summarization with Constrained Markov Decision Process](https://mlsanity.com/paper/2108.03405/) (2021)

## Built on by (most cited first, 7 of 7)

- [Detecting and Mitigating Hallucinations in Multilingual Summarisation](https://mlsanity.com/paper/2305.13632/) (2023)
- [Contrastive Hierarchical Discourse Graph for Scientific Document Summarization](https://mlsanity.com/paper/2306.00177/) (2023)
- [Flexible and Adaptable Summarization via Expertise Separation](https://mlsanity.com/paper/2406.05360/) (2024)
- [Overview of the BioLaySumm 2023 Shared Task on Lay Summarization of Biomedical Research Articles](https://mlsanity.com/paper/2309.17332/) (2023)
- [Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?](https://mlsanity.com/paper/2401.10415/) (2024)
- [Understanding Domain Learning in Language Models Through Subpopulation Analysis](https://mlsanity.com/paper/2210.12553/) (2022)
- [Investigating Consistency in Query-Based Meeting Summarization: A Comparative Study of Different Embedding Methods](https://mlsanity.com/paper/2402.06907/) (2024)

## Abstract (arXiv)

We argue that disentangling content selection from the budget used to cover salient content improves the performance and applicability of abstractive summarizers. Our method, FactorSum, does this disentanglement by factorizing summarization into two steps through an energy function: (1) generation of abstractive summary views; (2) combination of these views into a final summary, following a budget and content guidance. This guidance may come from different sources, including from an advisor model such as BART or BigBird, or in oracle mode -- from the reference. This factorization achieves significantly higher ROUGE scores on multiple benchmarks for long document summarization, namely PubMed, arXiv, and GovReport. Most notably, our model is effective for domain adaptation. When trained only on PubMed samples, it achieves a 46.29 ROUGE-1 score on arXiv, which indicates a strong performance due to more flexible budget adaptation and content selection less dependent on domain-specific textual structure.