Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large language models achieve strong performance on many language tasks, yet it remains unclear whether they integrate world knowledge with syntactic structure in a human-like, structure-sensitive way during ambiguity resolution. We test this question in Turkish prenominal relative-clause attachment ambiguities, where the same surface string permits high attachment (HA) or low attachment (LA). We construct ambiguous items that keep the syntactic configuration fixed and ensure both parses remain pragmatically possible, while graded event plausibility selectively favors High Attachment vs. Low Attachment. The contrasts are validated with independent norming ratings. In a speeded forced-choice comprehension experiment, humans show a large, correctly directed plausibility effect. We then evaluate Turkish and multilingual LLMs in a parallel preference-based setup that compares matched HA/LA continuations via mean per-token log-probability. Across models, plausibility-driven shifts are weak, unstable, or reversed. The results suggest that, in the tested models, plausibility information does not guide attachment preferences as reliably as it does in human judgments, and they highlight Turkish RC attachment as a useful cross-linguistic diagnostic beyond broad benchmarks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 44 references that are in the map. First posted on arXiv 6 Apr 2026.
Builds on
16 of 44 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2025Gemma 3 Technical Reportcited by 1,686
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
- 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
- 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
- 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
- 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
- 2024Forcing Diffuse Distributions out of Language Modelscited by 48
- 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
- 2022Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguitiescited by 30
- 2021Multilingual Language Models Predict Human Reading Behaviorcited by 29
3 more
- 2025Tokenization Standards for Linguistic Integrity: Turkish as a Benchmarkcited by 8
- 2025Dougal Dil .Icslemede Tokenizasyon Standartlari ve "Olcc"um"u: T"urkcce "Uzerinden B"uy"uk Dil Modellerinin Karcsilacstirmali Analizicited by 4
- 2025Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?cited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026STRIVE: Probing Reasoning Limits in Graded Plausibility Generation and Evaluation
- 2026Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
- 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
- 2026Tracing the ongoing emergence of human-like reasoning in Large Language Models
- 2026Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning
- 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- 2025The Dog the Cat Chased Stumped the Model: Measuring When Language Models Abandon Structure for Shortcuts
- 2026Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning