# Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization

arXiv: https://arxiv.org/abs/2305.16784 · HTML: https://mlsanity.com/paper/2305.16784/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 78 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Analyzing the Structure of Attention in a Transformer Language Model](https://mlsanity.com/paper/1906.04284/) (2019)
- [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://mlsanity.com/paper/2112.07916/) (2021)
- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://mlsanity.com/paper/2210.09932/) (2022)
- [SG-Net: Syntax-Guided Machine Reading Comprehension](https://mlsanity.com/paper/1908.05147/) (2019)
- [Do Syntax Trees Help Pre-trained Transformers Extract Information?](https://mlsanity.com/paper/2008.09084/) (2020)
- [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022)
- [NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better](https://mlsanity.com/paper/2202.12024/) (2022)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [Neural Extractive Text Summarization with Syntactic Compression](https://mlsanity.com/paper/1902.00863/) (2019)
- [Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://mlsanity.com/paper/2104.08400/) (2021)
- [Improving Abstraction in Text Summarization](https://mlsanity.com/paper/1808.07913/) (2018)
- [DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsing](https://mlsanity.com/paper/2110.04518/) (2021)
- [HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information](https://mlsanity.com/paper/2203.09629/) (2022)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)

## Built on by (most cited first, 17 of 17)

- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [eRST: A Signaled Graph Theory of Discourse Relations and Organization](https://mlsanity.com/paper/2403.13560/) (2024)
- [SciNews: From Scholarly Complexities to Public Narratives -- A Dataset for Scientific News Report Generation](https://mlsanity.com/paper/2403.17768/) (2024)
- [RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization](https://mlsanity.com/paper/2405.00657/) (2024)
- [What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations](https://mlsanity.com/paper/2502.08279/) (2025)
- [Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization](https://mlsanity.com/paper/2502.06185/) (2025)
- [Multi-LLM Text Summarization](https://mlsanity.com/paper/2412.15487/) (2024)
- [Large Language Models as Zero-Shot Keyphrase Extractors: A Preliminary Empirical Study](https://mlsanity.com/paper/2312.15156/) (2023)
- [Explanatory Summarization with Discourse-Driven Planning](https://mlsanity.com/paper/2504.19339/) (2025)
- [Leveraging Discourse Structure for Extractive Meeting Summarization](https://mlsanity.com/paper/2405.11055/) (2024)
- [Sci-LoRA: Mixture of Scientific LoRAs for Cross-Domain Lay Paraphrasing](https://mlsanity.com/paper/2505.18867/) (2025)
- [Synthetic Data Augmentation for Cross-domain Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2503.20588/) (2025)
- [Disco-RAG: Discourse-Aware Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.04377/) (2026)
- [Extract-and-Abstract: Unifying Extractive and Abstractive Summarization within Single Encoder-Decoder Framework](https://mlsanity.com/paper/2409.11827/) (2024)
- [BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering](https://mlsanity.com/paper/2511.06183/) (2025)
- [Summarization is Not Dead Yet](https://mlsanity.com/paper/2606.08000/) (2026)

## Abstract (arXiv)

For text summarization, the role of discourse structure is pivotal in discerning the core content of a text. Regrettably, prior studies on incorporating Rhetorical Structure Theory (RST) into transformer-based summarization models only consider the nuclearity annotation, thereby overlooking the variety of discourse relation types. This paper introduces the 'RSTformer', a novel summarization model that comprehensively incorporates both the types and uncertainty of rhetorical relations. Our RST-attention mechanism, rooted in document-level rhetorical structure, is an extension of the recently devised Longformer framework. Through rigorous evaluation, the model proposed herein exhibits significant superiority over state-of-the-art models, as evidenced by its notable performance on several automatic metrics and human evaluation.