Not quite Sherlock Holmes: Language model predictions do not reliably differentiate impossible from improbable events
Not on the map. Why?
Abstract (arXiv)
Can language models reliably predict that possible events are more likely than merely improbable ones? By teasing apart possibility, typicality, and contextual relatedness, we show that despite the results of previous work, language models' ability to do this is far from robust. In fact, under certain conditions, all models tested - including Llama 3, Gemma 2, and Mistral NeMo - perform at worse-than-chance level, assigning higher probabilities to impossible sentences such as 'the car was given a parking ticket by the brake' than to merely unlikely sentences such as 'the car was given a parking ticket by the explorer'.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 39 of its 84 references that are in the map. First posted on arXiv 7 Jun 2025.
Builds on
39 of 84 references.
References · 39
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2018Neural Network Acceptability Judgmentscited by 958
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2017Dense-Captioning Events in Videoscited by 793
- 2017Deep Reinforcement Learning that Matterscited by 754
- 2024Yi: Open Foundation Models by 01.AIcited by 719
- 2018Annotation Artifacts in Natural Language Inference Datacited by 671
27 more
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
- 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
- 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
- 2021Few-shot Learning with Multilingual Language Modelscited by 278
- 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
- 2018Targeted Syntactic Evaluation of Language Modelscited by 242
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
- 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
- 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 142
- 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
- 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
- 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 73
- 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
- 2019Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approachescited by 65
- 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
- 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
- 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
- 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
- 2020Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Primingcited by 41
- 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
- 2019The Scientific Method in the Science of Machine Learningcited by 15
- 2023When Language Models Fall in Love: Animacy Processing in Transformer Language Modelscited by 12
- 2021Learning to Learn to be Right for the Right Reasonscited by 4
- 2022Methods for Estimating and Improving Robustness of Language Modelscited by 3
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.