# Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining

arXiv: https://arxiv.org/abs/2109.04080 · HTML: https://mlsanity.com/paper/2109.04080/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 9 Sep 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Domain Agnostic Learning with Disentangled Representations](https://mlsanity.com/paper/1904.12347/) (2019)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization](https://mlsanity.com/paper/1805.05271/) (2018)
- [TED: A Pretrained Unsupervised Summarization Model with Theme Modeling and Denoising](https://mlsanity.com/paper/2001.00725/) (2020)
- [Augmenting Neural Response Generation with Context-Aware Topical Attention](https://mlsanity.com/paper/1811.01063/) (2018)
- [Generative Adversarial Network for Abstractive Text Summarization](https://mlsanity.com/paper/1711.09357/) (2017)
- [Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling](https://mlsanity.com/paper/2012.07311/) (2020)
- [Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks](https://mlsanity.com/paper/2010.10044/) (2020)
- [Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders](https://mlsanity.com/paper/2012.07300/) (2020)
- [Exploring Domain Shift in Extractive Text Summarization](https://mlsanity.com/paper/1908.11664/) (2019)

## Built on by (most cited first, 19 of 19)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Representation Learning for Resource-Constrained Keyphrase Generation](https://mlsanity.com/paper/2203.08118/) (2022)
- [Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarization](https://mlsanity.com/paper/2204.04362/) (2022)
- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation](https://mlsanity.com/paper/2207.14467/) (2022)
- [Abstractive Meeting Summarization: A Survey](https://mlsanity.com/paper/2208.04163/) (2022)
- [DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization](https://mlsanity.com/paper/2212.10018/) (2022)
- [Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization](https://mlsanity.com/paper/2502.17328/) (2025)
- [Leveraging Non-dialogue Summaries for Dialogue Summarization](https://mlsanity.com/paper/2210.09474/) (2022)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning](https://mlsanity.com/paper/2305.12077/) (2023)
- [Towards Understanding Omission in Dialogue Summarization](https://mlsanity.com/paper/2211.07145/) (2022)
- [QUARTZ : QA-based Unsupervised Abstractive Refinement for Task-oriented Dialogue Summarization](https://mlsanity.com/paper/2509.26302/) (2025)
- [Heuristic-based Inter-training to Improve Few-shot Multi-perspective Dialog Summarization](https://mlsanity.com/paper/2203.15590/) (2022)
- [He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues](https://mlsanity.com/paper/2210.15462/) (2022)
- [What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization](https://mlsanity.com/paper/2505.12474/) (2025)
- [User Review Writing via Interview with Dialogue Systems](https://mlsanity.com/paper/2603.07070/) (2026)
- [Depth F1: Improving Evaluation of Cross-Domain Text Classification by Measuring Semantic Generalizability](https://mlsanity.com/paper/2406.14695/) (2024)

## Abstract (arXiv)

With the rapid increase in the volume of dialogue data from daily life, there is a growing demand for dialogue summarization. Unfortunately, training a large summarization model is generally infeasible due to the inadequacy of dialogue data with annotated summaries. Most existing works for low-resource dialogue summarization directly pretrain models in other domains, e.g., the news domain, but they generally neglect the huge difference between dialogues and conventional articles. To bridge the gap between out-of-domain pretraining and in-domain fine-tuning, in this work, we propose a multi-source pretraining paradigm to better leverage the external summary data. Specifically, we exploit large-scale in-domain non-summary data to separately pretrain the dialogue encoder and the summary decoder. The combined encoder-decoder model is then pretrained on the out-of-domain summary data using adversarial critics, aiming to facilitate domain-agnostic summarization. The experimental results on two public datasets show that with only limited training data, our approach achieves competitive performance and generalizes well in different dialogue scenarios.