mlsanityData to 7 Oct 2026

VisMin: Visual Minimal-Change Understanding

arXiv 2407.16772 ↗First posted 23 Jul 2024cs.CV cs.CL cs.LGcited by 23

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-language models (VLMs). Existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar captions given an image. In this paper, we introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin), which requires models to predict the correct image-caption match given two images and two captions. The image pair and caption pair contain minimal changes, i.e., only one aspect changes at a time from among the following: object, attribute, count, and spatial relation. These changes test the models' understanding of objects, attributes (such as color, material, shape), counts, and spatial relationships between objects. We built an automatic framework using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. We also generate a large-scale training dataset to finetune CLIP and Idefics2, showing significant improvements in fine-grained understanding across benchmarks and in CLIP's general image-text alignment. We release all resources, including the benchmark, training data, and finetuned model checkpoints, at https://vismin.net/.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 78 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

40 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2024Mixtral of Expertscited by 1,785
27 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  7. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  13. 2022Visual Spatial Reasoningcited by 288
  14. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  15. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  16. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  17. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  18. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  19. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  20. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  21. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  22. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  23. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  24. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  25. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  26. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  27. 2022Image Retrieval from Contextual Descriptionscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 11.

23 papers.

Sort:

By area:

  1. 2024Assessing and Learning Alignment of Unimodal Vision and Language Modelscited by 31
  2. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  3. 2024CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Datasetcited by 13
  4. 2024Progress-Aware Video Frame Captioningcited by 10
  5. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  6. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?cited by 6
  7. 2025The Promise of RL for Autoregressive Image Editingcited by 6
  8. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  9. 2025Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Datacited by 4
  10. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  11. 2026AI Research Agents Narrow Scientific Explorationcited by 2
  12. 2026Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimizationcited by 2
11 more
  1. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  2. 2026Self-Supervised Learning from Structural Invariancecited by 1
  3. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMscited by 1
  4. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  5. 2026FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Searchcited by 1
  6. 2025Spatial Knowledge Graph-Guided Multimodal Synthesiscited by 0
  7. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  8. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Modelscited by 0
  9. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  10. 2026DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cuescited by 0
  11. 2026MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  2. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  3. 2026VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
  4. 2026VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
  5. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  6. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations
  7. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations
  8. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models