mlsanityData to 7 Oct 2026

FOIL it! Find One mismatch between Image and Language caption

arXiv 1705.01359 ↗First posted 3 May 2017cs.CV cs.CL cs.MMcited by 95

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we aim to understand whether current language and vision (LaVi) models truly grasp the interaction between the two modalities. To this end, we propose an extension of the MSCOCO dataset, FOIL-COCO, which associates images with both correct and "foil" captions, that is, descriptions of the image that are highly similar to the original ones, but contain one single mistake ("foil word"). We show that current LaVi models fall into the traps of this data and perform badly on three tasks: a) caption classification (correct vs. foil); b) foil word detection; c) foil word correction. Humans, in contrast, have near-perfect performance on those tasks. We demonstrate that merely utilising language cues is not enough to model FOIL-COCO and that it challenges the state-of-the-art by requiring a fine-grained understanding of the relation between text and image.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 21; 2024: 22; 2025: 16; 2026: 15.

95 papers.

Sort:

By area:

  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  3. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  5. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  6. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  7. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  8. 2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
  9. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  10. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  11. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  12. 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
38 more
  1. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  2. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  3. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 44
  4. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 44
  5. 2022MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Taskscited by 42
  6. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  7. 2021EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matchingcited by 36
  8. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  9. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  10. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  11. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  12. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  13. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  14. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  15. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  16. 2021Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Casecited by 20
  17. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectivescited by 17
  18. 2024How and where does CLIP process negation?cited by 16
  19. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  20. 2021QACE: Asking Questions to Evaluate an Image Captioncited by 15
  21. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Modelscited by 15
  22. 2023Data Augmentations for Improved (Large) Language Model Generalizationcited by 14
  23. 2024BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cuescited by 13
  24. 2024HICEScore: A Hierarchical Metric for Image Captioning Evaluationcited by 12
  25. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11
  26. 2023Generating Enhanced Negatives for Training Language-Based Object Detectorscited by 11
  27. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 11
  28. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  29. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  30. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11
  31. 2021Multi-Modal Semantic Inconsistency Detection in Social Media News Postscited by 10
  32. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  33. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
  34. 2023Grounded Image Text Matching with Mismatched Relation Reasoningcited by 8
  35. 2023The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Modelscited by 8
  36. 2024ColorFoil: Investigating Color Blindness in Large Vision and Language Modelscited by 7
  37. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMscited by 7
  38. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7

50 most cited of 95 All 95 (JSON)