# A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages

arXiv: https://arxiv.org/abs/2004.03032 · HTML: https://mlsanity.com/paper/2004.03032/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 20 of 20)

- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [Visual Comparison of Language Model Adaptation](https://mlsanity.com/paper/2208.08176/) (2022)
- [Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5](https://mlsanity.com/paper/2410.11627/) (2024)
- [Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model](https://mlsanity.com/paper/2310.15113/) (2023)
- [A multilabel approach to morphosyntactic probing](https://mlsanity.com/paper/2104.08464/) (2021)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [Morphosyntactic probing of multilingual BERT models](https://mlsanity.com/paper/2306.06205/) (2023)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [Morph Call: Probing Morphosyntactic Content of Multilingual Transformers](https://mlsanity.com/paper/2104.12847/) (2021)
- [Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement](https://mlsanity.com/paper/2403.13754/) (2024)
- [Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis](https://mlsanity.com/paper/2406.14678/) (2024)
- [Are pre-trained text representations useful for multilingual and multi-dimensional language proficiency modeling?](https://mlsanity.com/paper/2102.12971/) (2021)
- [A Morphology-Based Investigation of Positional Encodings](https://mlsanity.com/paper/2404.04530/) (2024)
- [Derivational Morphology Reveals Analogical Generalization in Large Language Models](https://mlsanity.com/paper/2411.07990/) (2024)
- [Rule-based Morphological Inflection Improves Neural Terminology Translation](https://mlsanity.com/paper/2109.04620/) (2021)
- [IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages?](https://mlsanity.com/paper/2410.02611/) (2024)
- [Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation](https://mlsanity.com/paper/2109.12584/) (2021)
- [Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?](https://mlsanity.com/paper/2112.12356/) (2021)
- [CCAE: A Corpus of Chinese-based Asian Englishes](https://mlsanity.com/paper/2310.05381/) (2023)

## Abstract (arXiv)

This work describes experiments which probe the hidden representations of several BERT-style models for morphological content. The goal is to examine the extent to which discrete linguistic structure, in the form of morphological features and feature values, presents itself in the vector representations and attention distributions of pre-trained language models for five European languages. The experiments contained herein show that (i) Transformer architectures largely partition their embedding space into convex sub-regions highly correlated with morphological feature value, (ii) the contextualized nature of transformer embeddings allows models to distinguish ambiguous morphological forms in many, but not all cases, and (iii) very specific attention head/layer combinations appear to hone in on subject-verb agreement.