# Event knowledge in large language models: the gap between the impossible and the unlikely

arXiv: https://arxiv.org/abs/2212.01488 · HTML: https://mlsanity.com/paper/2212.01488/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 208 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)

## Built on by (most cited first, 30 of 45)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Prompting is not a substitute for probability measurements in large language models](https://mlsanity.com/paper/2305.13264/) (2023)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models](https://mlsanity.com/paper/2403.14859/) (2024)
- [TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks](https://mlsanity.com/paper/2403.09207/) (2024)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods](https://mlsanity.com/paper/2403.00998/) (2024)
- [Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale](https://mlsanity.com/paper/2510.24963/) (2025)
- [When Language Models Fall in Love: Animacy Processing in Transformer Language Models](https://mlsanity.com/paper/2310.15004/) (2023)
- [XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs](https://mlsanity.com/paper/2502.19737/) (2025)
- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [Do LLMs Really Adapt to Domains? An Ontology Learning Perspective](https://mlsanity.com/paper/2407.19998/) (2024)
- [Linguistic Interpretability of Transformer-based Language Models: a systematic review](https://mlsanity.com/paper/2504.08001/) (2025)
- [GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?](https://mlsanity.com/paper/2406.09464/) (2024)
- [Large Language Model probabilities cannot distinguish between possible and impossible language](https://mlsanity.com/paper/2509.15114/) (2025)
- [Understanding Natural Language Understanding Systems. A Critical Analysis](https://mlsanity.com/paper/2303.04229/) (2023)
- [Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence](https://mlsanity.com/paper/2411.07533/) (2024)
- [Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition](https://mlsanity.com/paper/2503.10037/) (2025)
- [Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis](https://mlsanity.com/paper/2510.25356/) (2025)
- [LangVAE and LangSpace: Building and Probing for Language Model VAEs](https://mlsanity.com/paper/2505.00004/) (2025)
- [Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting](https://mlsanity.com/paper/2505.09852/) (2025)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility](https://mlsanity.com/paper/2507.12553/) (2025)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [What Makes Language Models Good-enough?](https://mlsanity.com/paper/2406.03666/) (2024)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Shades of Zero: Distinguishing Impossibility from Inconceivability](https://mlsanity.com/paper/2502.20469/) (2025)
- [Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events](https://mlsanity.com/paper/2506.06808/) (2025)
- [Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model](https://mlsanity.com/paper/2302.08150/) (2023)

## Abstract (arXiv)

Word co-occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs' semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pre-trained LLMs (from 2018's BERT to 2023's MPT) assign higher likelihood to plausible descriptions of agent-patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n=1,215), we found that pre-trained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign higher likelihood to possible vs. impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely vs. unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow-up analyses, we show that (i) LLM scores are driven by both plausibility and surface-level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.