# Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans

arXiv: https://arxiv.org/abs/2602.14100 · HTML: https://mlsanity.com/paper/2602.14100/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding](https://mlsanity.com/paper/2010.04903/) (2020)
- [Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2412.05149/) (2024)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection](https://mlsanity.com/paper/1810.07125/) (2018)
- [CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages](https://mlsanity.com/paper/1706.09031/) (2017)
- [Pushing the Limits of Low-Resource Morphological Inflection](https://mlsanity.com/paper/1908.05838/) (2019)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection](https://mlsanity.com/paper/1906.01280/) (2019)
- [Inflecting when there's no majority: Limitations of encoder-decoder neural networks as cognitive models for German plurals](https://mlsanity.com/paper/2005.08826/) (2020)
- [Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models](https://mlsanity.com/paper/2104.06483/) (2021)
- [Single-Model Encoder-Decoder with Explicit Morphological Representation for Reinflection](https://mlsanity.com/paper/1606.00589/) (2016)
- [Neural Multi-Source Morphological Reinflection](https://mlsanity.com/paper/1612.06027/) (2016)
- [Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers](https://mlsanity.com/paper/2410.21013/) (2024)

## Built on by (most cited first, 2 of 2)

- [When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology](https://mlsanity.com/paper/2605.20558/) (2026)
- [Probing Character-level Transformers for the Spanish L-shaped Morphome](https://mlsanity.com/paper/2608.03452/) (2026)

## Abstract (arXiv)

Whether neural networks can serve as cognitive models of morphological learning remains an open question. Recent work has shown that encoder-decoder models can acquire irregular patterns, but evidence that they generalize these patterns like humans is mixed. We investigate this using the Spanish L-shaped morphome, where only the first-person singular indicative (e.g., textitpongo `I put') shares its stem with all subjunctive forms (e.g., textitponga, pongas) despite lacking apparent phonological, semantic, or syntactic motivation. We compare five encoder-decoder transformers varying along two dimensions: sequential vs. position-invariant positional encoding, and atomic vs. decomposed tag representations. Positional encoding proves decisive: position-invariant models recover the correct L-shaped paradigm clustering even when L-shaped verbs are scarce in training, whereas sequential positional encoding models only partially capture the pattern. Yet none of the models productively generalize this pattern to novel forms. Position-invariant models generalize the L-shaped stem across subjunctive cells but fail to extend it to the first-person singular indicative, producing a mood-based generalization rather than the L-shaped morphomic pattern. Humans do the opposite, generalizing preferentially to the first-person singular indicative over subjunctive forms. None of the models reproduce the human pattern, highlighting the gap between statistical pattern reproduction and morphological abstraction.