# A Systematic Study of Leveraging Subword Information for Learning Word Representations

arXiv: https://arxiv.org/abs/1904.07994 · HTML: https://mlsanity.com/paper/1904.07994/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 6 of 6)

- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [Subword Segmental Language Modelling for Nguni Languages](https://mlsanity.com/paper/2210.06525/) (2022)
- [Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation](https://mlsanity.com/paper/2403.07567/) (2024)
- [Search Intenion Network for Personalized Query Auto-Completion in E-Commerce](https://mlsanity.com/paper/2403.02609/) (2024)
- [HDND: Hierarchical Dynamic Neural Decoding for Multilingual Word/Character Retrieval from Non-Invasive Brain Recordings](https://mlsanity.com/paper/2609.24095/) (2026)
- [Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learning](https://mlsanity.com/paper/2507.04018/) (2025)

## Abstract (arXiv)

The use of subword-level information (e.g., characters, character n-grams, morphemes) has become ubiquitous in modern word representation learning. Its importance is attested especially for morphologically rich languages which generate a large number of rare words. Despite a steadily increasing interest in such subword-informed word representations, their systematic comparative analysis across typologically diverse languages and different tasks is still missing. In this work, we deliver such a study focusing on the variation of two crucial components required for subword-level integration into word representation models: 1) segmentation of words into subword units, and 2) subword composition functions to obtain final word representations. We propose a general framework for learning subword-informed word representations that allows for easy experimentation with different segmentation and composition components, also including more advanced techniques based on position embeddings and self-attention. Using the unified framework, we run experiments over a large number of subword-informed word representation configurations (60 in total) on 3 tasks (general and rare word similarity, dependency parsing, fine-grained entity typing) for 5 languages representing 3 language types. Our main results clearly indicate that there is no "one-sizefits-all" configuration, as performance is both language- and task-dependent. We also show that configurations based on unsupervised segmentation (e.g., BPE, Morfessor) are sometimes comparable to or even outperform the ones based on supervised word segmentation.