mlsanityData to 7 Oct 2026

Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Test

arXiv 2510.12463 ↗First posted 14 Oct 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

The linguistic abilities of Large Language Models are a matter of ongoing debate. This study contributes to this discussion by investigating model performance in a morphological generalization task that involves novel words. Using a multilingual adaptation of the Wug Test, six models were tested across four partially unrelated languages (Catalan, English, Greek, and Spanish) and compared with human speakers. The aim is to determine whether model accuracy approximates human competence and whether it is shaped primarily by linguistic complexity or by the size of the linguistic community, which affects the quantity of available training data. Consistent with previous research, the results show that the models are able to generalize morphological processes to unseen words with human-like accuracy. However, accuracy patterns align more closely with community size and data availability than with structural complexity, refining earlier claims in the literature. In particular, languages with larger speaker communities and stronger digital representation, such as Spanish and English, revealed higher accuracy than less-resourced ones like Catalan and Greek. Overall, our findings suggest that model behavior is mainly driven by the richness of linguistic resources rather than by sensitivity to grammatical complexity, reflecting a form of performance that resembles human linguistic competence only superficially.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 51 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

16 of 51 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  5. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  6. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  8. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  9. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  10. 2024LLMs as Models for Analogical Reasoningcited by 16
  11. 2023Evaluating Neural Language Models as Cognitive Models of Language Acquisitioncited by 15
  12. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
4 more
  1. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  2. 2023Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Modelcited by 9
  3. 2023Emergence of a phonological bias in ChatGPTcited by 8
  4. 2023How Well Do Large Language Models Understand Syntax? An Evaluation by Asking Natural Language Questionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.