# Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

arXiv: https://arxiv.org/abs/2302.05895 · HTML: https://mlsanity.com/paper/2302.05895/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 72 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [Discourse-Aware Neural Extractive Text Summarization](https://mlsanity.com/paper/1910.14142/) (2019)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure](https://mlsanity.com/paper/2004.05080/) (2020)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction](https://mlsanity.com/paper/2002.00737/) (2020)
- [A Deep Sequential Model for Discourse Parsing on Multi-Party Dialogues](https://mlsanity.com/paper/1812.00176/) (2018)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [Learning Structured Text Representations](https://mlsanity.com/paper/1705.09207/) (2017)
- [Multi-turn Response Selection using Dialogue Dependency Relations](https://mlsanity.com/paper/2010.01502/) (2020)
- [From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions](https://mlsanity.com/paper/1906.01958/) (2019)
- [DADgraph: A Discourse-aware Dialogue Graph Neural Network for Multiparty Dialogue Machine Reading Comprehension](https://mlsanity.com/paper/2104.12377/) (2021)
- [Is Everything in Order? A Simple Way to Order Sentences](https://mlsanity.com/paper/2104.07064/) (2021)
- [DisCoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detection](https://mlsanity.com/paper/2109.09777/) (2021)
- [RST Parsing from Scratch](https://mlsanity.com/paper/2105.10861/) (2021)
- [Improving Multi-Party Dialogue Discourse Parsing via Domain Integration](https://mlsanity.com/paper/2110.04526/) (2021)
- [Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !](https://mlsanity.com/paper/2012.02144/) (2020)
- [Predicting Discourse Trees from Transformer-based Neural Summarizers](https://mlsanity.com/paper/2104.07058/) (2021)
- [Evaluating Discourse in Structured Text Representations](https://mlsanity.com/paper/1906.01472/) (2019)
- [Predicting Discourse Structure using Distant Supervision from Sentiment](https://mlsanity.com/paper/1910.14176/) (2019)

## Built on by (most cited first, 4 of 4)

- [Llamipa: An Incremental Discourse Parser](https://mlsanity.com/paper/2406.18256/) (2024)
- [Discursive Circuits: How Do Language Models Understand Discourse Relations?](https://mlsanity.com/paper/2510.11210/) (2025)
- [Unsupervised Mutual Learning of Discourse Parsing and Topic Segmentation in Dialogue](https://mlsanity.com/paper/2405.19799/) (2024)
- [Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extraction](https://mlsanity.com/paper/2402.13906/) (2024)

## Abstract (arXiv)

Discourse processing suffers from data sparsity, especially for dialogues. As a result, we explore approaches to build discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple tasks for fine-tuning and show that the dialogue-tailored Sentence Ordering task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals achieve encouraging results on the STAC corpus, with F1 scores of 57.2 and 59.3 for unsupervised and semi-supervised methods, respectively. When restricted to projective trees, our scores improved to 63.3 and 68.1.