mlsanityData to 7 Oct 2026

Discourse Probing of Pretrained Language Models

arXiv 2104.05882 ↗First posted 13 Apr 2021cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse -- but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 16 of its 40 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

16 of 40 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  8. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  9. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 57
  10. 2020What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual modelscited by 35
  11. 2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
  12. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
4 more
  1. 2021Top-down Discourse Parsing via Sequence Labellingcited by 9
  2. 2020Examining the rhetorical capacities of neural language modelscited by 7
  3. 2019Deep Bidirectional Transformers for Relation Extraction without Supervisioncited by 5
  4. 2018Narrative Modeling with Memory Chains and Semantic Supervisioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 8; 2024: 6; 2025: 8; 2026: 2.

38 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  3. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  4. 2022Probing Pretrained Models of Source Codecited by 26
  5. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  6. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  7. 2021Pragmatic competence of pre-trained language models through the lens of discourse connectivescited by 16
  8. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representationscited by 12
  9. 2023A Survey on Zero Pronoun Translationcited by 9
  10. 2022SocioProbe: What, When, and Where Language Models Learn about Sociodemographicscited by 7
  11. 2024Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherencecited by 7
  12. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
26 more
  1. 2025Towards Transparent AI: A Survey on Explainable Large Language Modelscited by 7
  2. 2022Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Modelscited by 6
  3. 2022Evaluation Benchmarks for Spanish Sentence Representationscited by 6
  4. 2024Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalizationcited by 6
  5. 2022SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Modelscited by 5
  6. 2022On the data requirements of probingcited by 4
  7. 2023Discourse Structure Extraction from Pre-Trained and Fine-Tuned Language Models in Dialoguescited by 4
  8. 2025Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Setcited by 4
  9. 2021Quantifying the Task-Specific Information in Text-Based Classificationscited by 3
  10. 2022Probing with Noise: Unpicking the Warp and Weft of Embeddingscited by 3
  11. 2022Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluationcited by 3
  12. 2023Large Language Models Are Partially Primed in Pronoun Interpretationcited by 3
  13. 2025Towards Transparent AI: A Survey on Explainable Language Modelscited by 3
  14. 2025Geospatial Mechanistic Interpretability of Large Language Modelscited by 2
  15. 2022"No, they did not": Dialogue response dynamics in pre-trained language modelscited by 1
  16. 2023Which One Are You Referring To? Multimodal Object Identification in Situated Dialoguecited by 1
  17. 2023How Does Pretraining Improve Discourse-Aware Translation?cited by 1
  18. 2023Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attentioncited by 1
  19. 2025From Monolingual to Bilingual: Investigating Language Conditioning in Large Language Models for Psycholinguistic Taskscited by 1
  20. 2025WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectivescited by 1
  21. 2026Instructions Shape Production of Language, not Processingcited by 1
  22. 2022Towards Domain-Independent Supervised Discourse Parsing Through Gradient Boostingcited by 0
  23. 2024To Drop or Not to Drop? Predicting Argument Ellipsis Judgments: A Case Study in Japanesecited by 0
  24. 2024Probing Omissions and Distortions in Transformer-based RDF-to-Text Modelscited by 0
  25. 2025Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selectioncited by 0
  26. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0