# ViCo: Word Embeddings from Visual Co-occurrences

arXiv: https://arxiv.org/abs/1908.08527 · HTML: https://mlsanity.com/paper/1908.08527/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2019.

## Built on by (most cited first, 10 of 10)

- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval](https://mlsanity.com/paper/2208.09843/) (2022)
- [Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2111.07180/) (2021)
- [Align before Search: Aligning Ads Image to Text for Accurate Cross-Modal Sponsored Search](https://mlsanity.com/paper/2309.16141/) (2023)
- [XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding](https://mlsanity.com/paper/2204.07316/) (2022)
- [Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models](https://mlsanity.com/paper/2407.05061/) (2024)
- [From Coarse to Fine-grained Concept based Discrimination for Phrase Detection](https://mlsanity.com/paper/2112.03237/) (2021)
- [Temporal Slowness in Central Vision Drives Semantic Object Learning](https://mlsanity.com/paper/2602.04462/) (2026)

## Abstract (arXiv)

We propose to learn word embeddings from visual co-occurrences. Two words co-occur visually if both words apply to the same image or image region. Specifically, we extract four types of visual co-occurrences between object and attribute words from large-scale, textually-annotated visual databases like VisualGenome and ImageNet. We then train a multi-task log-bilinear model that compactly encodes word "meanings" represented by each co-occurrence type into a single visual word-vector. Through unsupervised clustering, supervised partitioning, and a zero-shot-like generalization analysis we show that our word embeddings complement text-only embeddings like GloVe by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone. We further evaluate our embeddings on five downstream applications, four of which are vision-language tasks. Augmenting GloVe with our embeddings yields gains on all tasks. We also find that random embeddings perform comparably to learned embeddings on all supervised vision-language tasks, contrary to conventional wisdom.