# Combining Language and Vision with a Multimodal Skip-gram Model

arXiv: https://arxiv.org/abs/1501.02598 · HTML: https://mlsanity.com/paper/1501.02598/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2015.

## Built on by (most cited first, 28 of 28)

- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [Does Vision-and-Language Pretraining Improve Lexical Grounding?](https://mlsanity.com/paper/2109.10246/) (2021)
- [MCSE: Multimodal Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2204.10931/) (2022)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)
- [Imagination-Augmented Natural Language Understanding](https://mlsanity.com/paper/2204.08535/) (2022)
- [Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination](https://mlsanity.com/paper/2210.12261/) (2022)
- [KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning](https://mlsanity.com/paper/2403.17486/) (2024)
- [ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality](https://mlsanity.com/paper/2110.06376/) (2021)
- [World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models](https://mlsanity.com/paper/2306.08685/) (2023)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training](https://mlsanity.com/paper/2104.07500/) (2021)
- [Visual Grounding Strategies for Text-Only Natural Language Processing](https://mlsanity.com/paper/2103.13942/) (2021)
- [Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations](https://mlsanity.com/paper/2305.13668/) (2023)
- [Understanding and Measuring Robustness of Multimodal Learning](https://mlsanity.com/paper/2112.12792/) (2021)
- [Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment](https://mlsanity.com/paper/2312.01592/) (2023)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Experiential Semantic Information and Brain Alignment: Are Multimodal Models Better than Language Models?](https://mlsanity.com/paper/2504.00942/) (2025)
- [Predicting emergent linguistic compositions through time: Syntactic frame extension via multimodal chaining](https://mlsanity.com/paper/2109.04652/) (2021)
- [Efficient Multi-Modal Embeddings from Structured Data](https://mlsanity.com/paper/2110.02577/) (2021)
- [Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers](https://mlsanity.com/paper/2410.13984/) (2024)
- [Sentiment Analysis of Fashion Related Posts in Social Media](https://mlsanity.com/paper/2111.07815/) (2021)
- [Do Trajectories Encode Verb Meaning?](https://mlsanity.com/paper/2206.11953/) (2022)
- [Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction](https://mlsanity.com/paper/2210.12905/) (2022)
- [Learning grounded word meaning representations on similarity graphs](https://mlsanity.com/paper/2109.03084/) (2021)
- [Judicial Intelligent Assistant System: Extracting Events from Divorce Cases to Detect Disputes for the Judge](https://mlsanity.com/paper/2303.16751/) (2023)
- [Multi-criteria recommendation systems to foster online grocery](https://mlsanity.com/paper/2312.08393/) (2023)

## Abstract (arXiv)

We extend the SKIP-GRAM model of Mikolov et al. (2013a) by taking visual information into account. Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora. However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly. The MMSKIP-GRAM models achieve good performance on a variety of semantic benchmarks. Moreover, since they propagate visual information to all words, we use them to improve image labeling and retrieval in the zero-shot setup, where the test concepts are never seen during model training. Finally, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.