# John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

arXiv: https://arxiv.org/abs/2106.01060 · HTML: https://mlsanity.com/paper/2106.01060/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 2 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020)
- [Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming](https://mlsanity.com/paper/2010.03010/) (2020)
- [Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Items](https://mlsanity.com/paper/1808.10627/) (2018)
- [Is Multilingual BERT Fluent in Language Generation?](https://mlsanity.com/paper/1910.03806/) (2019)
- [Investigating representations of verb bias in neural language models](https://mlsanity.com/paper/2010.02375/) (2020)
- [The Sensitivity of Language Models and Humans to Winograd Schema Perturbations](https://mlsanity.com/paper/2005.01348/) (2020)
- [Discourse structure interacts with reference but not syntax in neural language models](https://mlsanity.com/paper/2010.04887/) (2020)
- [Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA](https://mlsanity.com/paper/2009.08257/) (2020)

## Built on by (most cited first, 5 of 5)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025)
- [Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities](https://mlsanity.com/paper/2501.12980/) (2025)
- [Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English](https://mlsanity.com/paper/2504.09886/) (2025)

## Abstract (arXiv)

Some interpersonal verbs can implicitly attribute causality to either their subject or their object and are therefore said to carry an implicit causality (IC) bias. Through this bias, causal links can be inferred from a narrative, aiding language comprehension. We investigate whether pre-trained language models (PLMs) encode IC bias and use it at inference time. We find that to be the case, albeit to different degrees, for three distinct PLM architectures. However, causes do not always need to be implicit -- when a cause is explicitly stated in a subordinate clause, an incongruent IC bias associated with the verb in the main clause leads to a delay in human processing. We hypothesize that the temporary challenge humans face in integrating the two contradicting signals, one from the lexical semantics of the verb, one from the sentence-level semantics, would be reflected in higher error rates for models on tasks dependent on causal links. The results of our study lend support to this hypothesis, suggesting that PLMs tend to prioritize lexical patterns over higher-order signals.