# Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models

arXiv: https://arxiv.org/abs/2003.11743 · HTML: https://mlsanity.com/paper/2003.11743/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Model](https://mlsanity.com/paper/2406.15279/) (2024)
- [Mitigating Open-Vocabulary Caption Hallucinations](https://mlsanity.com/paper/2312.03631/) (2023)
- [OG-SGG: Ontology-Guided Scene Graph Generation. A Case Study in Transfer Learning for Telepresence Robotics](https://mlsanity.com/paper/2202.10201/) (2022)
- [Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring](https://mlsanity.com/paper/2107.00372/) (2021)
- [Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models](https://mlsanity.com/paper/2508.08926/) (2025)
- [Contrastive Semantic Similarity Learning for Image Captioning Evaluation with Intrinsic Auto-encoder](https://mlsanity.com/paper/2106.15312/) (2021)
- [ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation](https://mlsanity.com/paper/2403.01306/) (2024)

## Abstract (arXiv)

Image captioning models have been able to generate grammatically correct and human understandable sentences. However most of the captions convey limited information as the model used is trained on datasets that do not caption all possible objects existing in everyday life. Due to this lack of prior information most of the captions are biased to only a few objects present in the scene, hence limiting their usage in daily life. In this paper, we attempt to show the biased nature of the currently existing image captioning models and present a new image captioning dataset, Egoshots, consisting of 978 real life images with no captions. We further exploit the state of the art pre-trained image captioning and object recognition networks to annotate our images and show the limitations of existing works. Furthermore, in order to evaluate the quality of the generated captions, we propose a new image captioning metric, object based Semantic Fidelity (SF). Existing image captioning metrics can evaluate a caption only in the presence of their corresponding annotations; however, SF allows evaluating captions generated for images without annotations, making it highly useful for real life generated captions.