mlsanityData to 14 Oct 2026

Measuring Progress in Fine-grained Vision-and-Language Understanding

arXiv 2305.07558 ↗First posted 12 May 2023cs.CL cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

While pretraining on large-scale image-text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack "fine-grained" understanding, such as the ability to recognise relationships, verbs, and numbers in images. This has resulted in an increased interest in the community to either develop new benchmarks or models for such capabilities. To better understand and quantify progress in this direction, we investigate four competitive V&L models on four fine-grained benchmarks. Through our analysis, we find that X-VLM (Zeng et al., 2022) consistently outperforms other baselines, and that modelling innovations can impact performance more than scaling Web data, which even degrades performance sometimes. Through a deeper investigation of X-VLM, we highlight the importance of both novel losses and rich data sources for learning fine-grained skills. Finally, we inspect training dynamics, and discover that for some tasks, performance peaks early in training or significantly fluctuates, never converging.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 52 of its 70 references that are in the map. First posted on arXiv 12 May 2023.

Builds on

52 of 70 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  2. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  14. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  15. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  16. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  18. 2022Visual Spatial Reasoningcited by 293
  19. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  20. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  21. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  22. 2022A Unified Sequence Interface for Vision Taskscited by 152
  23. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  24. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  25. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  26. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  27. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  28. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  29. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  30. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  31. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  32. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 64
  33. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  34. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  35. 2022Gender Artifacts in Visual Datasetscited by 38
  36. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  37. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Modelscited by 30
  38. 2022Image Retrieval from Contextual Descriptionscited by 26

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 5; 2026: 1.

25 papers.

Sort:

By area:

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 59
  3. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  4. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  5. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  6. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  7. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 24
  8. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  9. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 22
  10. 2024HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoningcited by 18
  11. 2024How and where does CLIP process negation?cited by 16
  12. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
13 more
  1. 2025Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Conceptscited by 12
  2. 2024PartIR: Composing SPMD Partitioning Strategies for Machine Learningcited by 10
  3. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?cited by 9
  4. 2023The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Modelscited by 8
  5. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  6. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentationcited by 6
  7. 2025Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?cited by 4
  8. 2025Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgettingcited by 3
  9. 2024Renaissance: Investigating the Pretraining of Vision-Language Encoderscited by 2
  10. 2023The Scenario Refiner: Grounding subjects in images at the morphological levelcited by 1
  11. 2025Align Beyond Prompts: Evaluating World Knowledge Alignment in Text-to-Image Generationcited by 1
  12. 2024Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correctioncited by 0
  13. 2026TempViz: On the Evaluation of Temporal Knowledge in Text-to-Image Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  3. 2024Improving Fine-grained Visual Understanding in VLMs through Text-Only Training
  4. 2025Infusing fine-grained visual knowledge to Vision-Language Models
  5. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  6. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
  7. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  8. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis