# Defoiling Foiled Image Captions

arXiv: https://arxiv.org/abs/1805.06549 · HTML: https://mlsanity.com/paper/1805.06549/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2018.

## Built on by (most cited first, 9 of 9)

- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks](https://mlsanity.com/paper/2212.08158/) (2022)
- [How and where does CLIP process negation?](https://mlsanity.com/paper/2407.10488/) (2024)
- [The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models](https://mlsanity.com/paper/2310.02777/) (2023)
- [What are You Looking at? Modality Contribution in Multimodal Medical Deep Learning](https://mlsanity.com/paper/2503.01904/) (2025)
- [Quantifying Modality Contributions via Disentangling Multimodal Representations](https://mlsanity.com/paper/2511.19470/) (2025)
- [From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Making](https://mlsanity.com/paper/2609.05149/) (2026)

## Abstract (arXiv)

We address the task of detecting foiled image captions, i.e. identifying whether a caption contains a word that has been deliberately replaced by a semantically similar word, thus rendering it inaccurate with respect to the image being described. Solving this problem should in principle require a fine-grained understanding of images to detect linguistically valid perturbations in captions. In such contexts, encoding sufficiently descriptive image information becomes a key challenge. In this paper, we demonstrate that it is possible to solve this task using simple, interpretable yet powerful representations based on explicit object information. Our models achieve state-of-the-art performance on a standard dataset, with scores exceeding those achieved by humans on the task. We also measure the upper-bound performance of our models using gold standard annotations. Our analysis reveals that the simpler model performs well even without image information, suggesting that the dataset contains strong linguistic bias.