mlsanityData to 7 Oct 2026

African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification

arXiv 2406.14496 ↗First posted 20 Jun 2024cs.CV cs.CLcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent Large Vision-Language Models (LVLMs) demonstrate impressive abilities on numerous image understanding and reasoning tasks. The task of fine-grained object classification (e.g., distinction between textitanimal species), however, has been probed insufficiently, despite its downstream importance. We fill this evaluation gap by creating textttFOCI (Fine-grained Object ClassIfication), a difficult multiple-choice benchmark for fine-grained object classification, from existing object classification datasets: (1) multiple-choice avoids ambiguous answers associated with casting classification as open-ended QA task; (2) we retain classification difficulty by mining negative labels with a CLIP model. textttFOCIxspace complements five popular classification datasets with four domain-specific subsets from ImageNet-21k. We benchmark 12 public LVLMs on textttFOCI and show that it tests for a textitcomplementary skill to established image understanding and reasoning benchmarks. Crucially, CLIP models exhibit dramatically better performance than LVLMs. Since the image encoders of LVLMs come from these CLIP models, this points to inadequate alignment for fine-grained object distinction between the encoder and the LLM and warrants (pre)training data with more fine-grained annotation. We release our code at urlhttps://github.com/gregor-ge/FOCI-Benchmark.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 51 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

26 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
12 more
  1. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  2. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  3. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  5. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  6. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  8. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  9. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  10. 2024Stable LM 2 1.6B Technical Reportcited by 86
  11. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  12. 2020Concept Generalization in Visual Representation Learningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 11.

31 papers.

Sort:

By area:

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  2. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  3. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 17
  5. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  6. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  7. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Datacited by 12
  8. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  9. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  10. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 10
  11. 2026Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?cited by 8
  12. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
19 more
  1. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  2. 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
  3. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  4. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5
  5. 2025Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMscited by 5
  6. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 3
  7. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  8. 2025Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategiescited by 2
  9. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Modelscited by 2
  10. 2026Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understandingcited by 2
  11. 2025CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluationcited by 1
  12. 2025UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrievalcited by 1
  13. 2024TinyEmo: Scaling down Emotional Reasoning via Metric Projectioncited by 0
  14. 2025Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Modelscited by 0
  15. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
  16. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  17. 2026Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Gamescited by 0
  18. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  19. 2026HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  2. 2025Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models
  3. 2024Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models
  4. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  5. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
  6. 2025Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  8. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understanding