mlsanityData to 7 Oct 2026

Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models

arXiv 2204.04289 ↗First posted 8 Apr 2022cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

With a growing number of BERTology work analyzing different components of pre-trained language models, we extend this line of research through an in-depth analysis of discourse information in pre-trained and fine-tuned language models. We move beyond prior work along three dimensions: First, we describe a novel approach to infer discourse structures from arbitrarily long documents. Second, we propose a new type of analysis to explore where and how accurately intrinsic discourse is captured in the BERT and BART models. Finally, we assess how similar the generated structures are to a variety of baselines as well as their distribution within and between models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 37 of its 65 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

37 of 65 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Reformer: The Efficient Transformercited by 1,658
25 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  7. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  8. 2019DocBERT: BERT for Document Classificationcited by 68
  9. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  10. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  11. 2021Discourse Probing of Pretrained Language Modelscited by 38
  12. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  13. 2017Neural Discourse Structure for Text Categorizationcited by 24
  14. 2019From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentionscited by 22
  15. 2015Better Document-level Sentiment Analysis from RST Discourse Parsingcited by 20
  16. 2021Pragmatic competence of pre-trained language models through the lens of discourse connectivescited by 16
  17. 2021RST Parsing from Scratchcited by 13
  18. 2021Probing Multilingual Language Models for Discoursecited by 12
  19. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  20. 2021Demoting the Lead Bias in News Summarization via Alternating Adversarial Learningcited by 10
  21. 2021Top-down Discourse Parsing via Sequence Labellingcited by 9
  22. 2020MEGA RST Discourse Treebanks with Structure and Nuclearity from Scalable Distant Sentiment Supervisioncited by 8
  23. 2020Examining the rhetorical capacities of neural language modelscited by 7
  24. 2019Deep Bidirectional Transformers for Relation Extraction without Supervisioncited by 5
  25. 2020Unleashing the Power of Neural Discourse Parsers -- A Context and Structure Aware Approach Using Large Scale Pretrainingcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2.

6 papers.

  1. 2024Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifscited by 18
  2. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  3. 2024eRST: A Signaled Graph Theory of Discourse Relations and Organizationcited by 13
  4. 2023Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialoguescited by 4
  5. 2023Visual Analytics for Generative Transformer Modelscited by 2
  6. 2022Towards Domain-Independent Supervised Discourse Parsing Through Gradient Boostingcited by 0