# Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities

arXiv: https://arxiv.org/abs/2501.12980 · HTML: https://mlsanity.com/paper/2501.12980/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 105 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [How well do Large Language Models perform in Arithmetic tasks?](https://mlsanity.com/paper/2304.02015/) (2023)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Deep RNNs Encode Soft Hierarchical Syntax](https://mlsanity.com/paper/1805.04218/) (2018)
- [Sorting through the noise: Testing robustness of information processing in pre-trained language models](https://mlsanity.com/paper/2109.12393/) (2021)
- [Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning](https://mlsanity.com/paper/2301.09626/) (2023)
- [Investigating representations of verb bias in neural language models](https://mlsanity.com/paper/2010.02375/) (2020)
- [Discourse structure interacts with reference but not syntax in neural language models](https://mlsanity.com/paper/2010.04887/) (2020)
- [Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi](https://mlsanity.com/paper/2204.08669/) (2022)
- [Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution](https://mlsanity.com/paper/2206.11815/) (2022)
- [a survey on GPT-3](https://mlsanity.com/paper/2212.00857/) (2022)
- [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://mlsanity.com/paper/2310.15113/) (2023)
- [WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case](https://mlsanity.com/paper/2409.05653/) (2024)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)

## Built on by (most cited first, 1 of 1)

- [Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English](https://mlsanity.com/paper/2504.09886/) (2025)

## Abstract (arXiv)

In this paper, we compare data generated with mono- and multilingual LLMs spanning a range of model sizes with data provided by human participants in an experimental setting investigating well-established discourse biases. Beyond the comparison as such, we aim to develop a benchmark to assess the capabilities of LLMs with discourse biases as a robust proxy for more general discourse understanding capabilities. More specifically, we investigated Implicit Causality verbs, for which psycholinguistic research has found participants to display biases with regard to three phenomena: the establishment of (i) coreference relations (Experiment 1), (ii) coherence relations (Experiment 2), and (iii) the use of particular referring expressions (Experiments 3 and 4). With regard to coreference biases we found only the largest monolingual LLM (German Bloom 6.4B) to display more human-like biases. For coherence relation, no LLM displayed the explanation bias usually found for humans. For referring expressions, all LLMs displayed a preference for referring to subject arguments with simpler forms than to objects. However, no bias effect on referring expression was found, as opposed to recent studies investigating human biases.