mlsanityData to 14 Oct 2026

Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities

arXiv 2412.16418 ↗First posted 21 Dec 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With the rapid advancement of Multimodal Large Language Models (MLLMs), a variety of benchmarks have been introduced to evaluate their capabilities. While most evaluations have focused on complex tasks such as scientific comprehension and visual reasoning, little attention has been given to assessing their fundamental image classification abilities. In this paper, we address this gap by thoroughly revisiting the MLLMs with an in-depth analysis of image classification. Specifically, building on established datasets, we examine a broad spectrum of scenarios, from general classification tasks (e.g., ImageNet, ObjectNet) to more fine-grained categories such as bird and food classification. Our findings reveal that the most recent MLLMs can match or even outperform CLIP-style vision-language models on several datasets, challenging the previous assumption that MLLMs are bad at image classification citeVLMClassifier. To understand the factors driving this improvement, we conduct an in-depth analysis of the network architecture, data selection, and training recipe used in public MLLMs. Our results attribute this success to advancements in language models and the diversity of training data sources. Based on these observations, we further analyze and attribute the potential reasons to conceptual knowledge transfer and enhanced exposure of target concepts, respectively. We hope our findings will offer valuable insights for future research on MLLMs and their evaluation in image classification tasks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 56 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

13 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation23 Sep
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2024Qwen2 Technical Reportcited by 2,120
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  11. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  2. 2025How to Teach Large Multimodal Models New Skillscited by 11
  3. 2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 11
  4. 2025On Large Multimodal Models as Open-World Image Classifierscited by 8
  5. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  6. 2025U-ViLAR: Uncertainty-Aware Visual Localization for Autonomous Driving via Differentiable Association and Registrationcited by 5
  7. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  8. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  9. 2025Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resamplingcited by 2
  10. 2026Multimodal Large Language Models as Image Classifierscited by 2
  11. 2026Large Multimodal Models as General In-Context Classifierscited by 1
  12. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
3 more
  1. 2026Specificity-aware reinforcement learning for fine-grained open-world classificationcited by 0
  2. 2026Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Gamescited by 0
  3. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Why are Visually-Grounded Language Models Bad at Image Classification?
  2. 2024Rethinking VLMs and LLMs for Image Classification
  3. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions
  4. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  5. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification
  6. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  7. 2025Image Recognition with Vision and Language Embeddings of VLMs
  8. 2024From Training-Free to Adaptive: Empirical Insights into MLLMs' Understanding of Detection Information