mlsanityData to 14 Oct 2026

Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models

arXiv 2501.15140 ↗First posted 25 Jan 2025cs.CV cs.AI cs.CLcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multi-modal large language models (MLLMs) have shown remarkable abilities in various visual understanding tasks. However, MLLMs still struggle with fine-grained visual recognition (FGVR), which aims to identify subordinate-level categories from images. This can negatively impact more advanced capabilities of MLLMs, such as object-centric visual question answering and reasoning. In our study, we revisit three quintessential capabilities of MLLMs for FGVR, including object information extraction, category knowledge reserve, object-category alignment, and position of the root cause as a misalignment problem. To address this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase. We employ contrastive learning on object-attribute pairs and attribute-category pairs simultaneously and use examples from similar but incorrect categories as hard negatives, naturally bringing representations of visual objects and category names closer. Extensive evaluations across multiple popular FGVR datasets demonstrate that Finedefics outperforms existing MLLMs of comparable parameter sizes, showcasing its remarkable efficacy. The code is available at https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 53 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

44 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification20 Jun
  2. 2024What matters when building vision-language models?3 May
  3. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
28 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  9. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  10. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  11. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  13. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  14. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  16. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  17. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  18. 2014Part-based R-CNNs for Fine-grained Category Detectioncited by 109
  19. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  20. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  21. 2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
  22. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  23. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  24. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41
  25. 2023Learning Visual Representations via Language-Guided Samplingcited by 32
  26. 2023Bootstrapping Vision-Language Learning with Decoupled Language Pre-trainingcited by 28
  27. 2023TransHP: Image Classification with Hierarchical Promptingcited by 16
  28. 2021The Curious Layperson: Fine-Grained Image Recognition without Expert Labelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 15.

31 papers.

Sort:

By area:

  1. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  2. 2025CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Gamescited by 14
  3. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  4. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 11
  5. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  6. 2026Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?cited by 8
  7. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  8. 2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?cited by 6
  9. 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
  10. 2025Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMscited by 6
  11. 2025UniF2ace: A Unified Fine-grained Face Understanding and Generation Modelcited by 5
  12. 2025CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methodscited by 5
19 more
  1. 2025PositionIC: Unified Position and Identity Consistency for Image Customizationcited by 4
  2. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  3. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  4. 2025Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMscited by 2
  5. 2026Seeing as Experts Do: A Knowledge-Augmented Agent for Open-Set Fine-Grained Visual Understandingcited by 2
  6. 2025MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classificationcited by 1
  7. 2025UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrievalcited by 1
  8. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Groundingcited by 1
  9. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  10. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  11. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
  12. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  13. 2026RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMscited by 0
  14. 2026Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Gamescited by 0
  15. 2026ToolFG: Towards Well-Grounded Fine-Grained Image Classificationcited by 0
  16. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  17. 2026DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentationcited by 0
  18. 2026Subtoken Vision Transformer for Fine-grained Recognitioncited by 0
  19. 2026From Glance to Focus: Mitigating Recognition Laziness in MLLMs for Fine-Grained Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  2. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  3. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  4. 2025Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategies
  5. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  6. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  7. 2024Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
  8. 2025Same or Not? Enhancing Visual Perception in Vision-Language Models