Data to 14 Oct 2026

Why are Visually-Grounded Language Models Bad at Image Classification?

arXiv 2405.18415 ↗First posted 28 May 2024cs.CV cs.AI cs.CLcited by 102

Not on the map. Why?

Cited by 68 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

37 of 55 references.

References · 37

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2020Language Models are Few-Shot Learnerscited by 32,419
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2023GPT-4 Technical Reportcited by 18,396
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
2020Measuring Massive Multitask Language Understandingcited by 7,963
2023Visual Instruction Tuningcited by 7,869
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
2016Overcoming catastrophic forgetting in neural networkscited by 4,787
2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
2022Large Language Models are Zero-Shot Reasonerscited by 4,374

25 more

2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
2019Towards VQA Models That Can Readcited by 1,718
2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
2023DataComp: In search of the next generation of multimodal datasetscited by 644
2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
2024What matters when building vision-language models?cited by 326
2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
2023Stay on topic with Classifier-Free Guidancecited by 91
2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
2024When Do We Not Need Larger Vision Models?cited by 67
2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 35
2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 46; 2026: 39.

102 papers.

Sort:

By area:

2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
2025Latent Visual Reasoningcited by 104
2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 68
2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
2025AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Modelscited by 33
2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 31
2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 24
2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21
2025Lost in Embeddings: Information Loss in Vision-Language Modelscited by 20
2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 19
2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 19

38 more

2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 16
2024What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insightscited by 13
2024PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generationcited by 13
2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
2025FakeScope: Large Multimodal Expert Model for Transparent AI-Generated Image Forensicscited by 11
2026Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoningcited by 11
2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 10
2024An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMscited by 9
2025LogicQA: Logical Anomaly Detection with Vision Language Model Generated Questionscited by 9
2025On Large Multimodal Models as Open-World Image Classifierscited by 8
2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 7
2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
2026Self-Refining Vision Language Model for Robotic Failure Detection and Reasoningcited by 7
2026Reasoning-Driven Multimodal LLM for Domain Generalizationcited by 7
2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
2025LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition?cited by 6
2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
2025Single Domain Generalization for Few-Shot Counting via Universal Representation Matchingcited by 6
2025Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMscited by 6
2026Limited Linguistic Diversity in Embodied AI Datasetscited by 6
2025Test-Time Distillation for Continual Model Adaptationcited by 5
2025Adopting a human developmental visual diet yields robust, shape-based AI visioncited by 5
2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
2025Embedding-based Retrieval in Multimodal Content Moderationcited by 4
2025WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classificationcited by 4
2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
2026FakeVLM-R1: Internalizing Physical Laws via CoT for Synthetic Image Detectioncited by 4
2024VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Modelscited by 3
2025Contrastive Visual Data Augmentationcited by 3
2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 3
2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
2024DifAttack++: Query-Efficient Black-Box Adversarial Attack via Hierarchical Disentangled Feature Space in Cross-Domaincited by 2
2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Modelscited by 2
2025Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMscited by 2
2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Modelscited by 2

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way