# Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers

arXiv: https://arxiv.org/abs/2410.21013 · HTML: https://mlsanity.com/paper/2410.21013/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 76 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection](https://mlsanity.com/paper/1810.07125/) (2018)
- [CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages](https://mlsanity.com/paper/1706.09031/) (2017)
- [SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection](https://mlsanity.com/paper/2006.11572/) (2020)
- [Character-based Neural Machine Translation](https://mlsanity.com/paper/1603.00810/) (2016)
- [Exact Hard Monotonic Attention for Character-Level Transduction](https://mlsanity.com/paper/1905.06319/) (2019)
- [A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation](https://mlsanity.com/paper/1603.06147/) (2016)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [Morphological Inflection Generation Using Character Sequence to Sequence Learning](https://mlsanity.com/paper/1512.06110/) (2015)
- [Single-Model Encoder-Decoder with Explicit Morphological Representation for Reinflection](https://mlsanity.com/paper/1606.00589/) (2016)
- [Imitation Learning for Neural Morphological String Transduction](https://mlsanity.com/paper/1808.10701/) (2018)
- [Neural Multi-Source Morphological Reinflection](https://mlsanity.com/paper/1612.06027/) (2016)
- [Modeling morphology with Linear Discriminative Learning: considerations and design choices](https://mlsanity.com/paper/2106.07936/) (2021)
- [Is deeper always better? Replacing linear mappings with deep learning networks in the Discriminative Lexicon Model](https://mlsanity.com/paper/2410.04259/) (2024)
- [A Comprehensive Comparison of Neural Networks as Cognitive Models of Inflection](https://mlsanity.com/paper/2210.12321/) (2022)

## Built on by (most cited first, 5 of 5)

- [Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans](https://mlsanity.com/paper/2602.14100/) (2026)
- [When Irregularity Helps: A Subclass Analysis of Inductive Bias in Neural Morphology](https://mlsanity.com/paper/2605.20558/) (2026)
- [Transformers over-extend what humans underlearn: the case of Spanish L-shaped morphome](https://mlsanity.com/paper/2507.21556/) (2025)
- [Probing Character-level Transformers for the Spanish L-shaped Morphome](https://mlsanity.com/paper/2608.03452/) (2026)
- [Not All Irregularity Is Equal: Causally Isolating a Rare Failure Mode in Japanese Morphological Inflection](https://mlsanity.com/paper/2609.21179/) (2026)

## Abstract (arXiv)

Over the past decade, various studies have addressed how speakers solve the so-called `The Paradigm Cell Filling Problem' (PCFP) citepackerman2009parts across different languages. The PCFP addresses a fundamental question in morphological processing: how do speakers accurately generate inflected forms of words when presented with incomplete paradigms? This problem is particularly salient when modeling complex inflectional systems. We focus on Spanish verbal paradigms, where certain verbs follow an irregular L-shaped pattern, where the first-person singular present indicative stem matches the stem used throughout the present subjunctive mood. We formulate the problem as a morphological reinflection task. Specifically, we investigate the role of input frequency in the acquisition of regular versus irregular L-shaped patterns in transformer models. By systematically manipulating the input distributions and analyzing model behavior, we reveal four key findings: 1) Models perform better on L-shaped verbs compared to regular verbs, especially in uneven frequency conditions; 2) Robust primacy effects are observed, but no consistent recency effects; 3) Memorization becomes more prominent as the proportion of L-shaped verbs increases; 4) There is a tendency to regularize L-shaped verbs when their consonant alternation pairs are rare or absent in the training data.