# Predicting Discourse Trees from Transformer-based Neural Summarizers

arXiv: https://arxiv.org/abs/2104.07058 · HTML: https://mlsanity.com/paper/2104.07058/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Hierarchical Transformers for Multi-Document Summarization](https://mlsanity.com/paper/1905.13164/) (2019)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [Language Models are Open Knowledge Graphs](https://mlsanity.com/paper/2010.11967/) (2020)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [Learning Structured Text Representations](https://mlsanity.com/paper/1705.09207/) (2017)
- [From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions](https://mlsanity.com/paper/1906.01958/) (2019)
- [Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !](https://mlsanity.com/paper/2012.02144/) (2020)
- [Evaluating Discourse in Structured Text Representations](https://mlsanity.com/paper/1906.01472/) (2019)
- [Predicting Discourse Structure using Distant Supervision from Sentiment](https://mlsanity.com/paper/1910.14176/) (2019)
- [MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervision](https://mlsanity.com/paper/2011.03017/) (2020)

## Built on by (most cited first, 10 of 10)

- [Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs](https://mlsanity.com/paper/2402.10586/) (2024)
- [Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarization](https://mlsanity.com/paper/2305.16784/) (2023)
- [Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization](https://mlsanity.com/paper/2502.06185/) (2025)
- [Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation](https://mlsanity.com/paper/2112.05364/) (2021)
- [Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models](https://mlsanity.com/paper/2204.04289/) (2022)
- [Predicting Above-Sentence Discourse Structure using Distant Supervision from Topic Segmentation](https://mlsanity.com/paper/2112.06196/) (2021)
- [Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues](https://mlsanity.com/paper/2302.05895/) (2023)
- [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025)
- [W-RST: Towards a Weighted RST-style Discourse Framework](https://mlsanity.com/paper/2106.02658/) (2021)
- [Topic-driven Distant Supervision Framework for Macro-level Discourse Parsing](https://mlsanity.com/paper/2305.13755/) (2023)

## Abstract (arXiv)

Previous work indicates that discourse information benefits summarization. In this paper, we explore whether this synergy between discourse and summarization is bidirectional, by inferring document-level discourse trees from pre-trained neural summarizers. In particular, we generate unlabeled RST-style discourse trees from the self-attention matrices of the transformer model. Experiments across models and datasets reveal that the summarizer learns both, dependency- and constituency-style discourse information, which is typically encoded in a single head, covering long- and short-distance discourse dependencies. Overall, the experimental results suggest that the learned discourse information is general and transferable inter-domain.