# Learning Multi-Modal Word Representation Grounded in Visual Context

arXiv: https://arxiv.org/abs/1711.03483 · HTML: https://mlsanity.com/paper/1711.03483/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2017.

## Built on by (most cited first, 7 of 7)

- [MCSE: Multimodal Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2204.10931/) (2022)
- [Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination](https://mlsanity.com/paper/2210.12261/) (2022)
- [Universal Multimodal Representation for Language Understanding](https://mlsanity.com/paper/2301.03344/) (2023)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training](https://mlsanity.com/paper/2104.07500/) (2021)
- [Corpus-Based Paraphrase Detection Experiments and Review](https://mlsanity.com/paper/2106.00145/) (2021)
- [Learning grounded word meaning representations on similarity graphs](https://mlsanity.com/paper/2109.03084/) (2021)

## Abstract (arXiv)

Representing the semantics of words is a long-standing problem for the natural language processing community. Most methods compute word semantics given their textual context in large corpora. More recently, researchers attempted to integrate perceptual and visual features. Most of these works consider the visual appearance of objects to enhance word representations but they ignore the visual environment and context in which objects appear. We propose to unify text-based techniques with vision-based techniques by simultaneously leveraging textual and visual context to learn multimodal word embeddings. We explore various choices for what can serve as a visual context and present an end-to-end method to integrate visual context elements in a multimodal skip-gram model. We provide experiments and extensive analysis of the obtained results.