# Capacity Constraints and the Multilingual Penalty for Lexical Disambiguation

arXiv: https://arxiv.org/abs/2602.05035 · HTML: https://mlsanity.com/paper/2602.05035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 4 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models](https://mlsanity.com/paper/2401.10440/) (2024)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [An Isotropy Analysis in the Multilingual BERT Embedding Space](https://mlsanity.com/paper/2110.04504/) (2021)
- [RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)](https://mlsanity.com/paper/2105.13266/) (2021)
- [Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement](https://mlsanity.com/paper/2403.13754/) (2024)
- [Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis](https://mlsanity.com/paper/2406.14678/) (2024)
- [Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity](https://mlsanity.com/paper/2511.21974/) (2025)

## Abstract (arXiv)

Multilingual language models (LMs) sometimes under-perform their monolingual counterparts, possibly due to capacity limitations. We quantify this ``multilingual penalty'' for lexical disambiguation--a task requiring precise semantic representations and contextualization mechanisms--using controlled datasets of human relatedness judgments for ambiguous words in both English and Spanish. Comparing monolingual and multilingual LMs from the same families, we find consistently reduced performance in multilingual LMs. We then explore three potential capacity constraints: representational (reduced embedding isotropy), attentional (reduced attention to disambiguating cues), and vocabulary-related (increased multi-token segmentation). Multilingual LMs show some evidence of all three limitations; moreover, these factors statistically account for the variance formerly attributed to a model's multilingual status. These findings suggest both that multilingual LMs do suffer from multiple capacity constraints, and that these constraints correlate with reduced disambiguation performance.