# One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages

arXiv: https://arxiv.org/abs/2102.02585 · HTML: https://mlsanity.com/paper/2102.02585/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 4 Feb 2021.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [Correlating neural and symbolic representations of language](https://mlsanity.com/paper/1905.06401/) (2019)
- [New word analogy corpus for exploring embeddings of Czech words](https://mlsanity.com/paper/1608.00789/) (2016)

## Built on by (most cited first, 4 of 4)

- [Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation](https://mlsanity.com/paper/2310.03477/) (2023)
- [Deep Learning Approaches to Lexical Simplification: A Survey](https://mlsanity.com/paper/2305.12000/) (2023)
- [Simple Additions, Substantial Gains: Expanding Scripts, Languages, and Lineage Coverage in URIEL+](https://mlsanity.com/paper/2510.27183/) (2025)
- [How Important Is Tokenization in French Medical Masked Language Models?](https://mlsanity.com/paper/2402.15010/) (2024)

## Abstract (arXiv)

Unsupervised representation learning of words from large multilingual corpora is useful for downstream tasks such as word sense disambiguation, semantic text similarity, and information retrieval. The representation precision of log-bilinear fastText models is mostly due to their use of subword information. In previous work, the optimization of fastText's subword sizes has not been fully explored, and non-English fastText models were trained using subword sizes optimized for English and German word analogy tasks. In our work, we find the optimal subword sizes on the English, German, Czech, Italian, Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We then propose a simple n-gram coverage model and we show that it predicts better-than-default subword sizes on the Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We show that the optimization of fastText's subword sizes matters and results in a 14% improvement on the Czech word analogy task. We also show that expensive parameter optimization can be replaced by a simple n-gram coverage model that consistently improves the accuracy of fastText models on the word analogy tasks by up to 3% compared to the default subword sizes, and that it is within 1% accuracy of the optimal subword sizes.