# UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations

arXiv: https://arxiv.org/abs/1904.05521 · HTML: https://mlsanity.com/paper/1904.05521/

## Key facts

- Cited by 2 papers in the map (data through 13 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 2 of 2)

- [Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations](https://mlsanity.com/paper/2204.09268/) (2022)
- [Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs](https://mlsanity.com/paper/2505.10836/) (2025)

## Abstract (arXiv)

We propose Unified Visual-Semantic Embeddings (UniVSE) for learning a joint space of visual and textual concepts. The space unifies the concepts at different levels, including objects, attributes, relations, and full scenes. A contrastive learning approach is proposed for the fine-grained alignment from only image-caption pairs. Moreover, we present an effective approach for enforcing the coverage of semantic components that appear in the sentence. We demonstrate the robustness of Unified VSE in defending text-domain adversarial attacks on cross-modal retrieval tasks. Such robustness also empowers the use of visual cues to resolve word dependencies in novel sentences.