# Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data

arXiv: https://arxiv.org/abs/2103.05131 · HTML: https://mlsanity.com/paper/2103.05131/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts](https://mlsanity.com/paper/1710.06071/) (2017)
- [Neural Summarization by Extracting Sentences and Words](https://mlsanity.com/paper/1603.07252/) (2016)
- [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://mlsanity.com/paper/1611.06607/) (2016)
- [Unsupervised Abstractive Meeting Summarization with Multi-Sentence Compression and Budgeted Submodular Maximization](https://mlsanity.com/paper/1805.05271/) (2018)
- [A Hierarchical Neural Autoencoder for Paragraphs and Documents](https://mlsanity.com/paper/1506.01057/) (2015)

## Built on by (most cited first, 4 of 4)

- [Learning Disentangled Textual Representations via Statistical Measures of Similarity](https://mlsanity.com/paper/2205.03589/) (2022)
- [shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation](https://mlsanity.com/paper/2306.03264/) (2023)
- [Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering](https://mlsanity.com/paper/2404.16192/) (2024)
- [CSTRL: Context-Driven Sequential Transfer Learning for Abstractive Radiology Report Summarization](https://mlsanity.com/paper/2503.05750/) (2025)

## Abstract (arXiv)

Interleaved texts, where posts belonging to different threads occur in a sequence, commonly occur in online chat posts, so that it can be time-consuming to quickly obtain an overview of the discussions. Existing systems first disentangle the posts by threads and then extract summaries from those threads. A major issue with such systems is error propagation from the disentanglement component. While end-to-end trainable summarization system could obviate explicit disentanglement, such systems require a large amount of labeled data. To address this, we propose to pretrain an end-to-end trainable hierarchical encoder-decoder system using synthetic interleaved texts. We show that by fine-tuning on a real-world meeting dataset (AMI), such a system out-performs a traditional two-step system by 22%. We also compare against transformer models and observed that pretraining with synthetic data both the encoder and decoder outperforms the BertSumExtAbs transformer model which pretrains only the encoder on a large dataset.