# Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes

arXiv: https://arxiv.org/abs/1511.07067 · HTML: https://mlsanity.com/paper/1511.07067/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2015.

## Built on by (most cited first, 9 of 9)

- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)
- [GOAL: Global-local Object Alignment Learning](https://mlsanity.com/paper/2503.17782/) (2025)
- [Kiki or Bouba? Sound Symbolism in Vision-and-Language Models](https://mlsanity.com/paper/2310.16781/) (2023)
- [Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training](https://mlsanity.com/paper/2104.07500/) (2021)
- [Repo2Vec: A Comprehensive Embedding Approach for Determining Repository Similarity](https://mlsanity.com/paper/2107.05112/) (2021)
- [The Lean Data Scientist: Recent Advances towards Overcoming the Data Bottleneck](https://mlsanity.com/paper/2211.07959/) (2022)
- [Adversarial Robustness of Visual Dialog](https://mlsanity.com/paper/2207.02639/) (2022)
- [Word Embedding Techniques for Classification of Star Ratings](https://mlsanity.com/paper/2504.13653/) (2025)

## Abstract (arXiv)

We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world. For instance, although "eats" and "stares at" seem unrelated in text, they share semantics visually. When people are eating something, they also tend to stare at the food. Grounding diverse relations like "eats" and "stares at" into vision remains challenging, despite recent progress in vision. We note that the visual grounding of words depends on semantics, and not the literal pixels. We thus use abstract scenes created from clipart to provide the visual grounding. We find that the embeddings we learn capture fine-grained, visually grounded notions of semantic relatedness. We show improvements over text-only word embeddings (word2vec) on three tasks: common-sense assertion classification, visual paraphrasing and text-based image retrieval. Our code and datasets are available online.