# Multimodal Word Distributions

arXiv: https://arxiv.org/abs/1704.08424 · HTML: https://mlsanity.com/paper/1704.08424/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2017.

## Built on by (most cited first, 12 of 12)

- [MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model](https://mlsanity.com/paper/2210.05335/) (2022)
- [ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More](https://mlsanity.com/paper/2403.12534/) (2024)
- [SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts](https://mlsanity.com/paper/2104.08809/) (2021)
- [Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping](https://mlsanity.com/paper/2404.00974/) (2024)
- [Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications](https://mlsanity.com/paper/2103.15330/) (2021)
- [Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations](https://mlsanity.com/paper/2204.09268/) (2022)
- [Subspace Representations for Soft Set Operations and Sentence Similarities](https://mlsanity.com/paper/2210.13034/) (2022)
- [TaxoBell: Gaussian Box Embeddings for Self-Supervised Taxonomy Expansion](https://mlsanity.com/paper/2601.09633/) (2026)
- [Multi-facet Universal Schema](https://mlsanity.com/paper/2103.15339/) (2021)
- [Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings](https://mlsanity.com/paper/2204.02710/) (2022)
- [Word Sense Induction with Knowledge Distillation from BERT](https://mlsanity.com/paper/2304.10642/) (2023)
- [Mixtures Closest to a Given Measure: A Semidefinite Programming Approach](https://mlsanity.com/paper/2509.22879/) (2025)

## Abstract (arXiv)

Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.