mlsanityData to 14 Oct 2026

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation

arXiv 2504.14988 ↗First posted 21 Apr 2025cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception capabilities, garnering significant attention. While numerous evaluation studies have emerged, assessing LVLMs both holistically and on specialized tasks, fine-grained image tasks-fundamental to computer vision-remain largely unexplored. To fill this gap, we introduce a comprehensive fine-grained evaluation benchmark, i.e., FG-BMK, comprising 1.01 million questions and 0.33 million images. Our evaluation systematically examines LVLMs from both human-oriented and machine-oriented perspectives, focusing on their semantic recognition and fine-grained feature representation capabilities. Through extensive experiments on twelve representative LVLMs/VLMs, we uncover key findings regarding the influence of training paradigms, modality alignment, perturbation susceptibility, and fine-grained category reasoning on task performance. This work provides critical insights into the limitations of current LVLMs and offers guidance for future data construction and model design in the development of more advanced LVLMs. Our code is open-source and available at https://github.com/SEU-VIPGroup/FG-BMK.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 58 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

13 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness19 Feb
  2. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification20 Jun
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 10

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  8. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  9. 2020Products-10K: A Large-scale Product Recognition Datasetcited by 42
  10. 2022MetaFormer: A Unified Meta Framework for Fine-Grained Recognitioncited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  4. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  5. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  6. 2026Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understandingcited by 2
  7. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  8. 2026Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequencescited by 1
  9. 2025Large Language Models Facilitate Vision Reflection in Image Classificationcited by 0
  10. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
  11. 2026Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Modelscited by 0
  12. 2026RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignmentcited by 0
7 more
  1. 2026FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisitioncited by 0
  2. 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenescited by 0
  3. 2026Reliability-Prioritized Fine-Grained Generation in Multimodal Largecited by 0
  4. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  5. 2026HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understandingcited by 0
  6. 2026Tanawpamana: Benchmarking Vision-Language Models on Philippine Cultural Heritage Sitescited by 0
  7. 2026VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  2. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  5. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  6. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models
  7. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly
  8. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning