# Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge

arXiv: https://arxiv.org/abs/2107.10922 · HTML: https://mlsanity.com/paper/2107.10922/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 42 references that are in the map.
- First posted on arXiv 22 Jul 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [Modeling Semantic Plausibility by Injecting World Knowledge](https://mlsanity.com/paper/1804.00619/) (2018)
- [Are Elephants Bigger than Butterflies? Reasoning about Sizes of Objects](https://mlsanity.com/paper/1602.00753/) (2016)
- [Can a Gorilla Ride a Camel? Learning Semantic Plausibility from Text](https://mlsanity.com/paper/1911.05689/) (2019)
- [SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition](https://mlsanity.com/paper/1906.02123/) (2019)
- [A Structured Distributional Model of Sentence Meaning and Processing](https://mlsanity.com/paper/1906.07280/) (2019)
- [Measuring Thematic Fit with Distributional Feature Overlap](https://mlsanity.com/paper/1707.05967/) (2017)

## Built on by (most cited first, 15 of 15)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models](https://mlsanity.com/paper/2405.09605/) (2024)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models](https://mlsanity.com/paper/2403.14859/) (2024)
- [EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Models](https://mlsanity.com/paper/2305.15268/) (2023)
- [Understanding Natural Language Understanding Systems. A Critical Analysis](https://mlsanity.com/paper/2303.04229/) (2023)
- [Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition](https://mlsanity.com/paper/2503.10037/) (2025)
- [Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios](https://mlsanity.com/paper/2209.07760/) (2022)
- [A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement](https://mlsanity.com/paper/2404.04035/) (2024)
- [Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPT](https://mlsanity.com/paper/2310.13824/) (2023)
- [Stress Test for BERT and Deep Models: Predicting Words from Italian Poetry](https://mlsanity.com/paper/2302.09303/) (2023)
- [Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representation](https://mlsanity.com/paper/2410.15173/) (2024)
- [Modelling Adjectival Modification Effects on Semantic Plausibility](https://mlsanity.com/paper/2507.21828/) (2025)

## Abstract (arXiv)

Prior research has explored the ability of computational models to predict a word semantic fit with a given predicate. While much work has been devoted to modeling the typicality relation between verbs and arguments in isolation, in this paper we take a broader perspective by assessing whether and to what extent computational approaches have access to the information about the typicality of entire events and situations described in language (Generalized Event Knowledge). Given the recent success of Transformers Language Models (TLMs), we decided to test them on a benchmark for the textitdynamic estimation of thematic fit. The evaluation of these models was performed in comparison with SDM, a framework specifically designed to integrate events in sentence meaning representations, and we conducted a detailed error analysis to investigate which factors affect their behavior. Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge, and their predictions often depend on surface linguistic features, such as frequent words, collocations and syntactic patterns, thereby showing sub-optimal generalization abilities.