mlsanityData to 14 Oct 2026

Language-Guided Invariance Probing of Vision-Language Models

arXiv 2511.13494 ↗First posted 17 Nov 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent vision-language models (VLMs) such as CLIP, OpenCLIP, EVA02-CLIP and SigLIP achieve strong zero-shot performance, but it is unclear how reliably they respond to controlled linguistic perturbations. We introduce Language-Guided Invariance Probing (LGIP), a benchmark that measures (i) invariance to meaning-preserving paraphrases and (ii) sensitivity to meaning-changing semantic flips in image-text matching. Using 40k MS COCO images with five human captions each, we automatically generate paraphrases and rule-based flips that alter object category, color or count, and summarize model behavior with an invariance error, a semantic sensitivity gap and a positive-rate statistic. Across nine VLMs, EVA02-CLIP and large OpenCLIP variants lie on a favorable invariance-sensitivity frontier, combining low paraphrase-induced variance with consistently higher scores for original captions than for their flipped counterparts. In contrast, SigLIP and SigLIP2 show much larger invariance error and often prefer flipped captions to the human descriptions, especially for object and color edits. These failures are largely invisible to standard retrieval metrics, indicating that LGIP provides a model-agnostic diagnostic for the linguistic robustness of VLMs beyond conventional accuracy scores.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 13 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

11 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Probing Visual Language Priors in VLMs31 Dec

Other references · 10

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  9. 2022Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shiftcited by 24
  10. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarkscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Consistency in Vision Language Models
  2. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  3. 2023Scalable Performance Analysis for Vision-Language Models
  4. 2026What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
  5. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
  6. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  7. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervision
  8. 2024ComAlign: Compositional Alignment in Vision-Language Models