# The Case for Perspective in Multimodal Datasets

arXiv: https://arxiv.org/abs/2205.10902 · HTML: https://mlsanity.com/paper/2205.10902/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 22 May 2022.

## Builds on (most cited first, 7 of 7)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description](https://mlsanity.com/paper/1710.07177/) (2017)
- [Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings](https://mlsanity.com/paper/1603.09188/) (2016)

## Built on by (most cited first, 1 of 1)

- [FutureVision: A methodology for the investigation of future cognition](https://mlsanity.com/paper/2502.01597/) (2025)

## Abstract (arXiv)

This paper argues in favor of the adoption of annotation practices for multimodal datasets that recognize and represent the inherently perspectivized nature of multimodal communication. To support our claim, we present a set of annotation experiments in which FrameNet annotation is applied to the Multi30k and the Flickr 30k Entities datasets. We assess the cosine similarity between the semantic representations derived from the annotation of both pictures and captions for frames. Our findings indicate that: (i) frame semantic similarity between captions of the same picture produced in different languages is sensitive to whether the caption is a translation of another caption or not, and (ii) picture annotation for semantic frames is sensitive to whether the image is annotated in presence of a caption or not.