mlsanityData to 7 Oct 2026

Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model

arXiv 2310.15113 ↗First posted 23 Oct 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. However, there have been relatively few systematic inquiries into the linguistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ignore the remarkable ability of humans to generalize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of human language, like morphology. Here, we close these gaps by conducting the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages (specifically, English, German, Tamil, and Turkish). We apply a version of Berko's (1958) wug test to ChatGPT, using novel, uncontaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English. Overall, our results -- through the lens of morphology -- cast a new light on the linguistic capabilities of ChatGPT, suggesting that claims of human-like language skills are premature and misleading.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 28 of its 74 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

28 of 74 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  6. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  7. 2022Data Contamination: From Memorization to Exploitationcited by 234
  8. 2019Assessing BERT's Syntactic Abilitiescited by 191
  9. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 189
  10. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  11. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  12. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
16 more
  1. 2020Applying the Transformer to Character-level Transductioncited by 43
  2. 2018The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflectioncited by 41
  3. 2017CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languagescited by 40
  4. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  5. 2018Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debatecited by 27
  6. 2020A Systematic Analysis of Morphological Content in BERT Models for Multiple Languagescited by 20
  7. 2021(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performancecited by 20
  8. 2020Inflecting when there's no majority: Limitations of encoder-decoder neural networks as cognitive models for German pluralscited by 13
  9. 2020Unsupervised Morphological Paradigm Completioncited by 8
  10. 2016Neural Multi-Source Morphological Reinflectioncited by 7
  11. 2020The Paradigm Discovery Problemcited by 6
  12. 2017Paradigm Completion for Derivational Morphologycited by 5
  13. 2017Context-Aware Prediction of Derivational Word-formscited by 4
  14. 2019Morphological Segmentation Inside-Outcited by 4
  15. 2022CaMEL: Case Marker Extraction without Labelscited by 3
  16. 2021Not quite there yet: Combining analogical patterns and encoder-decoder networks for cognitively plausible inflectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024Large corpora and large language models: a replicable method for automating grammatical annotationcited by 5
  2. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?cited by 3
  3. 2025Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Testcited by 2
  4. 2025Implicit Causality-biases in humans and LLMs as a tool for benchmarking LLM discourse capabilitiescited by 1
  5. 2025Can AI mimic the human ability to define neologisms?cited by 1
  6. 2025Talking with Oompa Loompas: A novel framework for evaluating linguistic acquisition of LLM agentscited by 1
  7. 2026Multilingual Large Language Models do not comprehend all natural languages to equal degreescited by 1
  8. 2025Quantification and object perception in Multimodal Large Language Models and human linguistic cognitioncited by 0
  9. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Modelscited by 0