# Evaluating Discourse Cohesion in Pre-trained Language Models

arXiv: https://arxiv.org/abs/2503.06137 · HTML: https://mlsanity.com/paper/2503.06137/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering](https://mlsanity.com/paper/1908.08167/) (2019)
- [Probing What Different NLP Tasks Teach Machines about Function Word Comprehension](https://mlsanity.com/paper/1904.11544/) (2019)
- [DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion](https://mlsanity.com/paper/1902.10526/) (2019)
- [Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations](https://mlsanity.com/paper/1909.00142/) (2019)
- [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://mlsanity.com/paper/2301.02724/) (2023)
- [Shallow Discourse Annotation for Chinese TED Talks](https://mlsanity.com/paper/2003.04032/) (2020)
- [Multi-Label Classification for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2406.04461/) (2024)
- [Leveraging Hierarchical Prototypes as the Verbalizer for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2411.14880/) (2024)
- [Quantitative Discourse Cohesion Analysis of Scientific Scholarly Texts using Multilayer Networks](https://mlsanity.com/paper/2205.07532/) (2022)

## Built on by (most cited first, 1 of 1)

- [MetaXCR: Reinforcement-Based Meta-Transfer Learning for Cross-Lingual Commonsense Reasoning](https://mlsanity.com/paper/2503.06531/) (2025)

## Abstract (arXiv)

Large pre-trained neural models have achieved remarkable success in natural language process (NLP), inspiring a growing body of research analyzing their ability from different aspects. In this paper, we propose a test suite to evaluate the cohesive ability of pre-trained language models. The test suite contains multiple cohesion phenomena between adjacent and non-adjacent sentences. We try to compare different pre-trained language models on these phenomena and analyze the experimental results,hoping more attention can be given to discourse cohesion in the future.