# Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?

arXiv: https://arxiv.org/abs/2503.10838 · HTML: https://mlsanity.com/paper/2503.10838/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [Do large language models resemble humans in language use?](https://mlsanity.com/paper/2303.08014/) (2023)
- [Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation](https://mlsanity.com/paper/2212.10140/) (2022)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment](https://mlsanity.com/paper/2005.00165/) (2020)
- [Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategy](https://mlsanity.com/paper/2501.15167/) (2025)
- [SLING: Sino Linguistic Evaluation of Large Language Models](https://mlsanity.com/paper/2210.11689/) (2022)
- [AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://mlsanity.com/paper/2302.00762/) (2023)
- [AmbigNLG: Addressing Task Ambiguity in Instruction for NLG](https://mlsanity.com/paper/2402.17717/) (2024)
- [Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles](https://mlsanity.com/paper/2404.08949/) (2024)
- [Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization](https://mlsanity.com/paper/2305.14917/) (2023)

## Built on by (most cited first, 3 of 3)

- [Correct-Detect: Balancing Performance and Ambiguity Through the Lens of Coreference Resolution in LLMs](https://mlsanity.com/paper/2509.14456/) (2025)
- [Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not](https://mlsanity.com/paper/2604.04825/) (2026)
- [Investigating Syntactic Biases in Multilingual Transformers with RC Attachment Ambiguities in Italian and English](https://mlsanity.com/paper/2504.09886/) (2025)

## Abstract (arXiv)

This study explores how recent large language models (LLMs) navigate relative clause attachment {ambiguity} and use world knowledge biases for disambiguation in six typologically diverse languages: English, Chinese, Japanese, Korean, Russian, and Spanish. We describe the process of creating a novel dataset -- MultiWho -- for fine-grained evaluation of relative clause attachment preferences in ambiguous and unambiguous contexts. Our experiments with three LLMs indicate that, contrary to humans, LLMs consistently exhibit a preference for local attachment, displaying limited responsiveness to syntactic variations or language-specific attachment patterns. Although LLMs performed well in unambiguous cases, they rigidly prioritized world knowledge biases, lacking the flexibility of human language processing. These findings highlight the need for more diverse, pragmatically nuanced multilingual training to improve LLMs' handling of complex structures and human-like comprehension.