# Implicit causality in GPT-2: a case study

arXiv: https://arxiv.org/abs/2212.04348 · HTML: https://mlsanity.com/paper/2212.04348/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Trading Off Diversity and Quality in Natural Language Generation](https://mlsanity.com/paper/2004.10450/) (2020)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Underreporting of errors in NLG output, and what to do about it](https://mlsanity.com/paper/2108.01182/) (2021)
- [Discourse structure interacts with reference but not syntax in neural language models](https://mlsanity.com/paper/2010.04887/) (2020)
- [Machine Reading, Fast and Slow: When Do Models "Understand" Language?](https://mlsanity.com/paper/2209.07430/) (2022)
- [Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning](https://mlsanity.com/paper/2106.01207/) (2021)

## Built on by (most cited first, 2 of 2)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilities](https://mlsanity.com/paper/2501.12980/) (2025)

## Abstract (arXiv)

This case study investigates the extent to which a language model (GPT-2) is able to capture native speakers' intuitions about implicit causality in a sentence completion task. We first reproduce earlier results (showing lower surprisal values for pronouns that are congruent with either the subject or object, depending on which one corresponds to the implicit causality bias of the verb), and then examine the effects of gender and verb frequency on model performance. Our second study examines the reasoning ability of GPT-2: is the model able to produce more sensible motivations for why the subject VERBed the object if the verbs have stronger causality biases? We also developed a methodology to avoid human raters being biased by obscenities and disfluencies generated by the model.