mlsanityData to 7 Oct 2026

On the Limitations of Vision-Language Models in Understanding Image Transforms

arXiv 2503.09837 ↗First posted 12 Mar 2025cs.CV cs.AI cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

23 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Teaching CLIP to Count to Ten23 Feb

Other references · 22

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  4. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  5. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  6. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 197
  7. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  8. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedbackcited by 153
  9. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  10. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 119
  11. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  12. 2023Custom-Edit: Text-Guided Image Editing with Customized Diffusion Modelscited by 54
10 more
  1. 2023ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systemscited by 28
  2. 2022Fine-grained Image Editing by Pixel-wise Guidance Using Diffusion Modelscited by 18
  3. 2024OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learningcited by 14
  4. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  5. 2024CLIP-EBC: CLIP Can Count Accurately through Enhanced Blockwise Classificationcited by 12
  6. 2024Exploring the Impact of AI-generated Image Tools on Professional and Non-professional Users in the Art and Design Fieldscited by 10
  7. 2024Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIPcited by 9
  8. 2024Finetuning CLIP to Reason about Pairwise Differencescited by 6
  9. 2024Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?cited by 1
  10. 2024No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard Negatives via CLIP Knowledge and LLMscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2026The Spatial Blindspot of Vision-Language Modelscited by 5
  2. 2025RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotationcited by 4
  3. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  4. 2026AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMscited by 1
  5. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVAcited by 1
  6. 2025A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understandingcited by 0
  7. 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMscited by 0
  8. 2025Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Taskscited by 0
  9. 2025KidsArtBench: Multi-Dimensional Children's Art Evaluation with Attribute-Aware MLLMscited by 0
  10. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Taskscited by 0
  11. 2026Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspectivecited by 0
  12. 2026Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Introduction to Vision-Language Modeling
  2. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  3. 2023Vision Language Transformers: A Survey
  4. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  5. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities
  6. 2025A Survey on Efficient Vision-Language Models
  7. 2024Probing Visual Language Priors in VLMs
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone