# Evaluating Morphological Compositional Generalization in Large Language Models

arXiv: https://arxiv.org/abs/2410.12656 · HTML: https://mlsanity.com/paper/2410.12656/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 73 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts](https://mlsanity.com/paper/2306.04528/) (2023)
- [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://mlsanity.com/paper/2005.00955/) (2020)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Linguistic generalization and compositionality in modern artificial neural networks](https://mlsanity.com/paper/1904.00157/) (2019)
- [Are All Languages Equally Hard to Language-Model?](https://mlsanity.com/paper/1806.03743/) (2018)
- [How Do In-Context Examples Affect Compositional Generalization?](https://mlsanity.com/paper/2305.04835/) (2023)
- [The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection](https://mlsanity.com/paper/1810.07125/) (2018)
- [SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection](https://mlsanity.com/paper/2006.11572/) (2020)
- [Probing Linguistic Systematicity](https://mlsanity.com/paper/2005.04315/) (2020)
- [Why don't people use character-level machine translation?](https://mlsanity.com/paper/2110.08191/) (2021)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [Poro 34B and the Blessing of Multilinguality](https://mlsanity.com/paper/2404.01856/) (2024)
- [On the Complexity and Typology of Inflectional Morphological Systems](https://mlsanity.com/paper/1807.02747/) (2018)
- [Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction](https://mlsanity.com/paper/1909.02855/) (2019)
- [How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure](https://mlsanity.com/paper/2311.04900/) (2023)
- [Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation](https://mlsanity.com/paper/2305.07005/) (2023)

## Built on by (most cited first, 19 of 19)

- [TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2506.13487/) (2025)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Tokens with Meaning: A Hybrid Tokenization Approach for Turkish](https://mlsanity.com/paper/2508.14292/) (2025)
- [Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs](https://mlsanity.com/paper/2505.16134/) (2025)
- [UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?](https://mlsanity.com/paper/2508.11260/) (2025)
- [Vision Language Models are Confused Tourists](https://mlsanity.com/paper/2511.17004/) (2025)
- [Can AI mimic the human ability to define neologisms?](https://mlsanity.com/paper/2502.14900/) (2025)
- [A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans](https://mlsanity.com/paper/2507.12039/) (2025)
- [CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Setting](https://mlsanity.com/paper/2507.21257/) (2025)
- [Artificial Phantasia: Emergent Mental Imagery in Large Language Models](https://mlsanity.com/paper/2509.23108/) (2025)
- [Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs](https://mlsanity.com/paper/2603.15773/) (2026)
- [AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity](https://mlsanity.com/paper/2509.14171/) (2025)
- [Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis](https://mlsanity.com/paper/2602.10881/) (2026)
- [Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models](https://mlsanity.com/paper/2603.16718/) (2026)
- [AGEL-Comp: A Neuro-Symbolic Framework for Compositional Generalization in Interactive Agents](https://mlsanity.com/paper/2604.26522/) (2026)
- [YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models](https://mlsanity.com/paper/2609.10153/) (2026)
- [Binding Phoneme Identity to Position in Neural Models and the Human Brain](https://mlsanity.com/paper/2610.15843/) (2026)
- [Tracing Morphology, Semantics, and Divergence in Language Acquisition](https://mlsanity.com/paper/2610.19041/) (2026)
- [From Copying to Composition: How Language Models Develop In-Context Learning](https://mlsanity.com/paper/2610.19113/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabilities remain questionable, raising doubts about whether these models learn language similarly to humans. While humans exhibit compositional generalization and linguistic creativity in language use, the extent to which LLMs replicate these abilities, particularly in morphology, is under-explored. In this work, we systematically investigate the morphological generalization abilities of LLMs through the lens of compositionality. We define morphemes as compositional primitives and design a novel suite of generative and discriminative tasks to assess morphological productivity and systematicity. Focusing on agglutinative languages such as Turkish and Finnish, we evaluate several state-of-the-art instruction-finetuned multilingual models, including GPT-4 and Gemini. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots, with performance declining sharply as morphological complexity increases. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.