# Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings

arXiv: https://arxiv.org/abs/1603.09188 · HTML: https://mlsanity.com/paper/1603.09188/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2016.

## Built on by (most cited first, 9 of 9)

- [From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding](https://mlsanity.com/paper/2304.00553/) (2023)
- [MultiSubs: A Large-scale Multimodal and Multilingual Dataset](https://mlsanity.com/paper/2103.01910/) (2021)
- [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023)
- [Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categories](https://mlsanity.com/paper/2308.10562/) (2023)
- [Predicting emergent linguistic compositions through time: Syntactic frame extension via multimodal chaining](https://mlsanity.com/paper/2109.04652/) (2021)
- [The Case for Perspective in Multimodal Datasets](https://mlsanity.com/paper/2205.10902/) (2022)
- [An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition](https://mlsanity.com/paper/2210.04933/) (2022)
- [HKUST at SemEval-2023 Task 1: Visual Word Sense Disambiguation with Context Augmentation and Visual Assistance](https://mlsanity.com/paper/2311.18273/) (2023)
- [Bridging Lexical Ambiguity and Vision: A Mini Review on Visual Word Sense Disambiguation](https://mlsanity.com/paper/2602.01193/) (2026)

## Abstract (arXiv)

We introduce a new task, visual sense disambiguation for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one that describes the action depicted in the image. Just as textual word sense disambiguation is useful for a wide range of NLP tasks, visual sense disambiguation can be useful for multimodal tasks such as image retrieval, image description, and text illustration. We introduce VerSe, a new dataset that augments existing multimodal datasets (COCO and TUHOI) with sense labels. We propose an unsupervised algorithm based on Lesk which performs visual sense disambiguation using textual, visual, or multimodal embeddings. We find that textual embeddings perform well when gold-standard textual annotations (object labels and image descriptions) are available, while multimodal embeddings perform well on unannotated images. We also verify our findings by using the textual and multimodal embeddings as features in a supervised setting and analyse the performance of visual sense disambiguation task. VerSe is made publicly available and can be downloaded at: https://github.com/spandanagella/verse.