# Pragmatic competence of pre-trained language models through the lens of discourse connectives

arXiv: https://arxiv.org/abs/2109.12951 · HTML: https://mlsanity.com/paper/2109.12951/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [Mining Discourse Markers for Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/1903.11850/) (2019)
- [Probing Multilingual Language Models for Discourse](https://mlsanity.com/paper/2106.04832/) (2021)

## Built on by (most cited first, 17 of 17)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it](https://mlsanity.com/paper/2205.03472/) (2022)
- [SEA-HELM: Southeast Asian Holistic Evaluation of Language Models](https://mlsanity.com/paper/2502.14301/) (2025)
- [BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2309.06085/) (2023)
- [Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom](https://mlsanity.com/paper/2404.19509/) (2024)
- [Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models](https://mlsanity.com/paper/2204.04289/) (2022)
- [Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs](https://mlsanity.com/paper/2506.04534/) (2025)
- [Representing Affect Information in Word Embeddings](https://mlsanity.com/paper/2209.10583/) (2022)
- [WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives](https://mlsanity.com/paper/2510.09556/) (2025)
- [The Linguistic Blind Spot of Value-Aligned Agency, Natural and Artificial](https://mlsanity.com/paper/2207.00868/) (2022)
- ["No, they did not": Dialogue response dynamics in pre-trained language models](https://mlsanity.com/paper/2210.02526/) (2022)
- [Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue](https://mlsanity.com/paper/2302.14680/) (2023)
- [Discourse over Discourse: The Need for an Expanded Pragmatic Focus in Conversational AI](https://mlsanity.com/paper/2304.14543/) (2023)
- [Expanding the Set of Pragmatic Considerations in Conversational AI](https://mlsanity.com/paper/2310.18435/) (2023)
- [Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs](https://mlsanity.com/paper/2602.02878/) (2026)
- [Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models](https://mlsanity.com/paper/2610.13794/) (2026)

## Abstract (arXiv)

As pre-trained language models (LMs) continue to dominate NLP, it is increasingly important that we understand the depth of language capabilities in these models. In this paper, we target pre-trained LMs' competence in pragmatics, with a focus on pragmatics relating to discourse connectives. We formulate cloze-style tests using a combination of naturally-occurring data and controlled inputs drawn from psycholinguistics. We focus on testing models' ability to use pragmatic cues to predict discourse connectives, models' ability to understand implicatures relating to connectives, and the extent to which models show humanlike preferences regarding temporal dynamics of connectives. We find that although models predict connectives reasonably well in the context of naturally-occurring data, when we control contexts to isolate high-level pragmatic cues, model sensitivity is much lower. Models also do not show substantial humanlike temporal preferences. Overall, the findings suggest that at present, dominant pre-training paradigms do not result in substantial pragmatic competence in our models.