mlsanityData to 7 Oct 2026

Benchmark Granularity and Model Robustness for Image-Text Retrieval

arXiv 2407.15239 ↗First posted 21 Jul 2024cs.CV cs.AI cs.IRcited by 5

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Image-Text Retrieval (ITR) systems are central to multimodal information access, with Vision-Language Models (VLMs) showing strong performance on standard benchmarks. However, these benchmarks predominantly rely on coarse-grained annotations, limiting their ability to reveal how models perform under real-world conditions, where query granularity varies. Motivated by this gap, we examine how dataset granularity and query perturbations affect retrieval performance and robustness across four architecturally diverse VLMs (ALIGN, AltCLIP, CLIP, and GroupViT). Using both standard benchmarks (MS-COCO, Flickr30k) and their fine-grained variants, we show that richer captions consistently enhance retrieval, especially in text-to-image tasks, where we observe an average improvement of 16.23%, compared to 6.44% in image-to-text. To assess robustness, we introduce a taxonomy of perturbations and conduct extensive experiments, revealing that while perturbations typically degrade performance, they can also unexpectedly improve retrieval, exposing nuanced model behaviors. Notably, word order emerges as a critical factor -- contradicting prior assumptions of model insensitivity to it. Our results highlight variation in model robustness and a dataset-dependent relationship between caption granularity and perturbation sensitivity and emphasize the necessity of evaluating models on datasets of varying granularity.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 83 references that are in the map. First posted on arXiv 21 Jul 2024.

Builds on

48 of 83 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
36 more
  1. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  2. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  5. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  6. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  7. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  8. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  9. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 132
  10. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  11. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 113
  12. 2020Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrievalcited by 98
  13. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  14. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  15. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  16. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  17. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  18. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  19. 2021What Context Features Can Transformer Language Models Use?cited by 49
  20. 2020Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoderscited by 47
  21. 2021Evaluating the Robustness of Retrieval Pipelines with Query Variation Generatorscited by 33
  22. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  23. 2019Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variablescited by 26
  24. 2021Extending CLIP for Category-to-image Retrieval in E-commercecited by 24
  25. 2022Analysing the Robustness of Dual Encoders for Dense Retrieval Against Misspellingscited by 23
  26. 2021Cross-Market Product Recommendationcited by 22
  27. 2022CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typoscited by 22
  28. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
  29. 2014Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotationcited by 16
  30. 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15
  31. 2024Multi-granular Adversarial Attacks against Black-box Neural Ranking Modelscited by 14
  32. 2023A Study on FGSM Adversarial Training for Neural Retrievalcited by 10
  33. 2023Rethinking Benchmarks for Cross-modal Image-text Retrievalcited by 10
  34. 2021Controlled Caption Generation for Images Through Adversarial Attackscited by 9
  35. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-trainingcited by 7
  36. 2021A Linguistic Study on Relevance Modeling in Information Retrievalcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Image-Seeking Intent Prediction for Cross-Device Product Searchcited by 0
  2. 2026Achieving Text-based Person Retrieval with Any Granularitycited by 0
  3. 2026MRBench: A Comprehensive Benchmark for Human Motion-Text Retrievalcited by 0
  4. 2026Generative Universal Multimodal Retrieval with Dual-role Identifierscited by 0
  5. 2026MINER: Multi-crop INference-time Enhancement for Rare-Object Retrieval with Frozen Dual Encoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift
  2. 2026A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
  3. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  4. 2024FiCo-ITR: bridging fine-grained and coarse-grained image-text retrieval for comparative performance analysis
  5. 2025On the Brittleness of CLIP Text Encoders
  6. 2025A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback
  7. 2026Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models
  8. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models