# (Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance

arXiv: https://arxiv.org/abs/2108.05682 · HTML: https://mlsanity.com/paper/2108.05682/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks](https://mlsanity.com/paper/1807.07545/) (2018)
- [The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection](https://mlsanity.com/paper/1910.11493/) (2019)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [UniMorph 2.0: Universal Morphology](https://mlsanity.com/paper/1810.11101/) (2018)
- [The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection](https://mlsanity.com/paper/1810.07125/) (2018)
- [CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages](https://mlsanity.com/paper/1706.09031/) (2017)
- [SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection](https://mlsanity.com/paper/2006.11572/) (2020)
- [Split and Rephrase](https://mlsanity.com/paper/1707.06971/) (2017)
- [Split and Rephrase: Better Evaluation and a Stronger Baseline](https://mlsanity.com/paper/1805.01035/) (2018)
- [Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models](https://mlsanity.com/paper/2104.06483/) (2021)
- [Rethinking End-to-End Evaluation of Decomposable Tasks: A Case Study on Spoken Language Understanding](https://mlsanity.com/paper/2106.15065/) (2021)
- [Unsupervised Morphological Paradigm Completion](https://mlsanity.com/paper/2005.00970/) (2020)
- [Minimal Supervision for Morphological Inflection](https://mlsanity.com/paper/2104.08512/) (2021)

## Built on by (most cited first, 20 of 20)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Evaluating Morphological Compositional Generalization in Large Language Models](https://mlsanity.com/paper/2410.12656/) (2024)
- [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://mlsanity.com/paper/2310.15113/) (2023)
- [CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models](https://mlsanity.com/paper/2305.14214/) (2023)
- [Exploring the Landscape of Natural Language Processing Research](https://mlsanity.com/paper/2307.10652/) (2023)
- [Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers](https://mlsanity.com/paper/2410.21013/) (2024)
- [Morphological Reinflection with Multiple Arguments: An Extended Annotation schema and a Georgian Case Study](https://mlsanity.com/paper/2203.08527/) (2022)
- [Understanding Compositional Data Augmentation in Typologically Diverse Morphological Inflection](https://mlsanity.com/paper/2305.13658/) (2023)
- [An Investigation of Noise in Morphological Inflection](https://mlsanity.com/paper/2305.16581/) (2023)
- [Morphology Without Borders: Clause-Level Morphology](https://mlsanity.com/paper/2202.12832/) (2022)
- [Morphological Inflection: A Reality Check](https://mlsanity.com/paper/2305.15637/) (2023)
- [Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans](https://mlsanity.com/paper/2602.14100/) (2026)
- [Exploring Linguistic Probes for Morphological Generalization](https://mlsanity.com/paper/2310.13686/) (2023)
- [MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek](https://mlsanity.com/paper/2607.28274/) (2026)
- [Morphological Inflection with Phonological Features](https://mlsanity.com/paper/2306.12581/) (2023)
- [OOVs in the Spotlight: How to Inflect them?](https://mlsanity.com/paper/2404.08974/) (2024)
- [Probing Subphonemes in Morphology Models](https://mlsanity.com/paper/2505.11297/) (2025)
- [Transformers over-extend what humans underlearn: the case of Spanish L-shaped morphome](https://mlsanity.com/paper/2507.21556/) (2025)
- [Flexing in 73 Languages: A Single Small Model for Multilingual Inflection](https://mlsanity.com/paper/2510.23114/) (2025)
- [Corpus Frequencies in Morphological Inflection: Do They Matter?](https://mlsanity.com/paper/2510.23131/) (2025)

## Abstract (arXiv)

In the domain of Morphology, Inflection is a fundamental and important task that gained a lot of traction in recent years, mostly via SIGMORPHON's shared-tasks. With average accuracy above 0.9 over the scores of all languages, the task is considered mostly solved using relatively generic neural seq2seq models, even with little data provided. In this work, we propose to re-evaluate morphological inflection models by employing harder train-test splits that will challenge the generalization capacity of the models. In particular, as opposed to the na"ive split-by-form, we propose a split-by-lemma method to challenge the performance on existing benchmarks. Our experiments with the three top-ranked systems on the SIGMORPHON's 2020 shared-task show that the lemma-split presents an average drop of 30 percentage points in macro-average for the 90 languages included. The effect is most significant for low-resourced languages with a drop as high as 95 points, but even high-resourced languages lose about 10 points on average. Our results clearly show that generalizing inflection to unseen lemmas is far from being solved, presenting a simple yet effective means to promote more sophisticated models.