mlsanityData to 7 Oct 2026

Predicting Discourse Trees from Transformer-based Neural Summarizers

arXiv 2104.07058 ↗First posted 14 Apr 2021cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Previous work indicates that discourse information benefits summarization. In this paper, we explore whether this synergy between discourse and summarization is bidirectional, by inferring document-level discourse trees from pre-trained neural summarizers. In particular, we generate unlabeled RST-style discourse trees from the self-attention matrices of the transformer model. Experiments across models and datasets reveal that the summarizer learns both, dependency- and constituency-style discourse information, which is typically encoded in a single head, covering long- and short-distance discourse dependencies. Overall, the experimental results suggest that the learned discourse information is general and transferable inter-domain.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

17 of 39 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2019Text Summarization with Pretrained Encoderscited by 553
  4. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  5. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  6. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  7. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  8. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  9. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  10. 2020Language Models are Open Knowledge Graphscited by 95
  11. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  12. 2017Learning Structured Text Representationscited by 24
5 more
  1. 2019From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentionscited by 22
  2. 2020Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !cited by 10
  3. 2019Evaluating Discourse in Structured Text Representationscited by 9
  4. 2019Predicting Discourse Structure using Distant Supervision from Sentimentcited by 8
  5. 2020MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervisioncited by 8

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 1; 2023: 3; 2024: 1; 2025: 2.

10 papers.

  1. 2024Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifscited by 18
  2. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  3. 2025Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarizationcited by 8
  4. 2021Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentationcited by 6
  5. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  6. 2021Predicting Above-Sentence Discourse Structure using Distant Supervision from Topic Segmentationcited by 5
  7. 2023Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialoguescited by 4
  8. 2025Joint Modeling of Entities and Discourse Relations for Coherence Assessmentcited by 2
  9. 2021W-RST: Towards a Weighted RST-style Discourse Frameworkcited by 1
  10. 2023Topic-driven Distant Supervision Framework for Macro-level Discourse Parsingcited by 0