mlsanityData to 14 Oct 2026

Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialogues

arXiv 2302.05895 ↗First posted 12 Feb 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Discourse processing suffers from data sparsity, especially for dialogues. As a result, we explore approaches to build discourse structures for dialogues, based on attention matrices from Pre-trained Language Models (PLMs). We investigate multiple tasks for fine-tuning and show that the dialogue-tailored Sentence Ordering task performs best. To locate and exploit discourse information in PLMs, we propose an unsupervised and a semi-supervised method. Our proposals achieve encouraging results on the STAC corpus, with F1 scores of 57.2 and 59.3 for unsupervised and semi-supervised methods, respectively. When restricted to projective trees, our scores improved to 63.3 and 68.1.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 12 Feb 2023.

Builds on

35 of 72 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2019How Can We Know What Language Models Know?cited by 959
  6. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  8. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  11. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  12. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 105
23 more
  1. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  2. 2020Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structurecited by 60
  3. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  4. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  5. 2018A Deep Sequential Model for Discourse Parsing on Multi-Party Dialoguescited by 38
  6. 2021Discourse Probing of Pretrained Language Modelscited by 38
  7. 2017Learning Structured Text Representationscited by 24
  8. 2020Multi-turn Response Selection using Dialogue Dependency Relationscited by 23
  9. 2019From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentionscited by 22
  10. 2021DADgraph: A Discourse-aware Dialogue Graph Neural Network for Multiparty Dialogue Machine Reading Comprehensioncited by 16
  11. 2021Is Everything in Order? A Simple Way to Order Sentencescited by 15
  12. 2021DisCoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detectioncited by 15
  13. 2021RST Parsing from Scratchcited by 14
  14. 2021Improving Multi-Party Dialogue Discourse Parsing via Domain Integrationcited by 14
  15. 2020Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help !cited by 10
  16. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  17. 2019Evaluating Discourse in Structured Text Representationscited by 9
  18. 2019Predicting Discourse Structure using Distant Supervision from Sentimentcited by 8
  19. 2020MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervisioncited by 8
  20. 2020Examining the rhetorical capacities of neural language modelscited by 7
  21. 2017Joint Modeling of Content and Discourse Relations in Dialoguescited by 6
  22. 2021Multi-tasking Dialogue Comprehension with Discourse Parsingcited by 6
  23. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Llamipa: An Incremental Discourse Parsercited by 4
  2. 2025Discursive Circuits: How Do Language Models Understand Discourse Relations?cited by 4
  3. 2024Unsupervised Mutual Learning of Discourse Parsing and Topic Segmentation in Dialoguecited by 3
  4. 2024Leveraging Collection-Wide Similarities for Unsupervised Document Structure Extractioncited by 2