# Probabilistic FastText for Multi-Sense Word Embeddings

arXiv: https://arxiv.org/abs/1806.02901 · HTML: https://mlsanity.com/paper/1806.02901/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2018.

## Built on by (most cited first, 25 of 25)

- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [SemGes: Semantics-aware Co-Speech Gesture Generation using Semantic Coherence and Relevance Learning](https://mlsanity.com/paper/2507.19359/) (2025)
- [DEPLAIN: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification](https://mlsanity.com/paper/2305.18939/) (2023)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [TIPS: Text-Induced Pose Synthesis](https://mlsanity.com/paper/2207.11718/) (2022)
- [CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering](https://mlsanity.com/paper/2310.09536/) (2023)
- [The Foundational Capabilities of Large Language Models in Predicting Postoperative Risks Using Clinical Notes](https://mlsanity.com/paper/2402.17493/) (2024)
- [Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication task](https://mlsanity.com/paper/2105.06546/) (2021)
- [BSpell: A CNN-Blended BERT Based Bangla Spell Checker](https://mlsanity.com/paper/2208.09709/) (2022)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [Matina: A Large-Scale 73B Token Persian Text Corpus](https://mlsanity.com/paper/2502.09188/) (2025)
- [Multi-Sense Embeddings for Language Models and Knowledge Distillation](https://mlsanity.com/paper/2504.06036/) (2025)
- [End-to-End Natural Language Understanding Pipeline for Bangla Conversational Agents](https://mlsanity.com/paper/2107.05541/) (2021)
- [Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations](https://mlsanity.com/paper/2204.09268/) (2022)
- [Constructing Colloquial Dataset for Persian Sentiment Analysis of Social Microblogs](https://mlsanity.com/paper/2306.12679/) (2023)
- [TaxoBell: Gaussian Box Embeddings for Self-Supervised Taxonomy Expansion](https://mlsanity.com/paper/2601.09633/) (2026)
- [EDS-MEMBED: Multi-sense embeddings based on enhanced distributional semantic structures via a graph walk over word senses](https://mlsanity.com/paper/2103.00232/) (2021)
- [Detecting Suicidality in Arabic Tweets Using Machine Learning and Deep Learning Techniques](https://mlsanity.com/paper/2309.00246/) (2023)
- [NushuRescue: Revitalization of the Endangered Nushu Language with AI](https://mlsanity.com/paper/2412.00218/) (2024)
- [Variational Gaussian Topic Model with Invertible Neural Projections](https://mlsanity.com/paper/2105.10095/) (2021)
- [Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Language](https://mlsanity.com/paper/2110.13032/) (2021)
- [Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings](https://mlsanity.com/paper/2204.02710/) (2022)
- [Automatic Annotation of Direct Speech in Written French Narratives](https://mlsanity.com/paper/2306.15634/) (2023)
- [Low-Resource Neural Machine Translation Using Recurrent Neural Networks and Transfer Learning: A Case Study on English-to-Igbo](https://mlsanity.com/paper/2504.17252/) (2025)

## Abstract (arXiv)

We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.