mlsanityData to 14 Oct 2026

Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

arXiv 2312.00081 ↗First posted 30 Nov 2023cs.CVcited by 46

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision language models (VLM) have demonstrated remarkable performance across various downstream tasks. However, understanding fine-grained visual-linguistic concepts, such as attributes and inter-object relationships, remains a significant challenge. While several benchmarks aim to evaluate VLMs in finer granularity, their primary focus remains on the linguistic aspect, neglecting the visual dimension. Here, we highlight the importance of evaluating VLMs from both a textual and visual perspective. We introduce a progressive pipeline to synthesize images that vary in a specific attribute while ensuring consistency in all other aspects. Utilizing this data engine, we carefully design a benchmark, SPEC, to diagnose the comprehension of object size, position, existence, and count. Subsequently, we conduct a thorough evaluation of four leading VLMs on SPEC. Surprisingly, their performance is close to random guess, revealing significant limitations. With this in mind, we propose a simple yet effective approach to optimize VLMs in fine-grained understanding, achieving significant improvements on SPEC without compromising the zero-shot performance. Results on two additional fine-grained benchmarks also show consistent improvements, further validating the transferability of our approach. Code and data are available at https://github.com/wjpoom/SPEC.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 45 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

39 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023Segment Anythingcited by 8,234
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
24 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2023Segment Anything in High Qualitycited by 376
  10. 2022Visual Spatial Reasoningcited by 293
  11. 2023A Survey on Video Diffusion Modelscited by 206
  12. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  13. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  14. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  15. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  16. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  17. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  18. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  19. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 55
  20. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  21. 2022X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusioncited by 49
  22. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  23. 2023Building an Open-Vocabulary Video CLIP Model with Better Architectures, Optimization and Datacited by 22
  24. 2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 24; 2026: 11.

46 papers.

Sort:

By area:

  1. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  2. 2025Vision-Language Models Do Not Understand Negationcited by 72
  3. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 59
  4. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  5. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  6. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 29
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  8. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  9. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  10. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 12
  11. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  12. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 12
34 more
  1. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  2. 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Modelscited by 10
  3. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  4. 2025SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentationcited by 9
  5. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  6. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 9
  7. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  8. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  9. 2025Diffusion Classifiers Understand Compositionality, but Conditions Applycited by 7
  10. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  11. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 6
  12. 2025Enhancing Vision Foundation Models via Multimodal Continual Pre-Trainingcited by 5
  13. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  14. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  15. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  16. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  17. 2025PARTONOMY: Large Multimodal Models with Part-Level Visual Understandingcited by 4
  18. 2025CIVET: Systematic Evaluation of Understanding in VLMscited by 4
  19. 2026Weakly Supervised Video Anomaly Detection with Anomaly-Connected Components and Intention Reasoningcited by 3
  20. 2025ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Modelscited by 2
  21. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  22. 2024Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstructioncited by 1
  23. 2025Can Text-to-Video Generation help Video-Language Alignment?cited by 1
  24. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinationscited by 1
  25. 2026Information Router for Mitigating Modality Dominance in Vision-Language Modelscited by 1
  26. 2026When Do Diffusion Models learn to Generate Multiple Objects?cited by 1
  27. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  28. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilitiescited by 0
  29. 2026ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Modelscited by 0
  30. 2026Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Modelscited by 0
  31. 2026CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objectscited by 0
  32. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  33. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0
  34. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  2. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  3. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  4. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  5. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
  6. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  7. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  8. 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models