# A Novel LLM-based Two-stage Summarization Approach for Long Dialogues

arXiv: https://arxiv.org/abs/2410.06520 · HTML: https://mlsanity.com/paper/2410.06520/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [Heterogeneous Graph Neural Networks for Extractive Document Summarization](https://mlsanity.com/paper/2004.12393/) (2020)
- [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Unsupervised Topic Segmentation of Meetings with BERT Embeddings](https://mlsanity.com/paper/2106.12978/) (2021)
- [Improving Unsupervised Dialogue Topic Segmentation with Utterance-Pair Coherence Scoring](https://mlsanity.com/paper/2106.06719/) (2021)
- [Paragraph-level Commonsense Transformers with Recurrent Memory](https://mlsanity.com/paper/2010.01486/) (2020)
- [Mind the Gap! Injecting Commonsense Knowledge for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00930/) (2022)
- [Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation](https://mlsanity.com/paper/2305.02747/) (2023)
- [GoSum: Extractive Summarization of Long Documents by Reinforcement Learning and Graph Organized discourse state](https://mlsanity.com/paper/2211.10247/) (2022)
- [Long Dialog Summarization: An Analysis](https://mlsanity.com/paper/2402.16986/) (2024)
- [Enhancing Semantic Understanding with Self-supervised Methods for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2209.00278/) (2022)

## Built on by (most cited first, 3 of 3)

- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [ArgCMV: An Argument Summarization Benchmark for the LLM-era](https://mlsanity.com/paper/2508.19580/) (2025)
- [From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applications](https://mlsanity.com/paper/2602.18443/) (2026)

## Abstract (arXiv)

Long document summarization poses a significant challenge in natural language processing due to input lengths that exceed the capacity of most state-of-the-art pre-trained language models. This study proposes a hierarchical framework that segments and condenses information from long documents, subsequently fine-tuning the processed text with an abstractive summarization model. Unsupervised topic segmentation methods identify semantically appropriate breakpoints. The condensation stage utilizes an unsupervised generation model to generate condensed data, and our current experiments employ ChatGPT(v3.5). The summarization stage fine-tunes the abstractive summarization model on the condensed data to generate the final results. This framework enables long documents to be processed on models even when the document length exceeds the model's maximum input size. The exclusion of the entire document from the summarization model reduces the time and computational resources required for training, making the framework suitable for contexts with constrained local computational resources.