mlsanityData to 14 Oct 2026

The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition

arXiv 2505.24840 ↗First posted 30 May 2025cs.CV cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper reveals that many open-source large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual recognition (e.g., recognizing Anemone Fish but not Vertebrate). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect because the VQA tasks improve the LLMs' hierarchical consistency more than the vision LLMs'. We conjecture that one cannot make open-source vision LLMs understand visual concepts hierarchically until LLMs possess corresponding taxonomy knowledge.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 58 of its 82 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

58 of 82 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  13. 2023Large Language Modelscited by 780
  14. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  15. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
  16. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  17. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  18. 2023Hyperbolic Image-Text Representationscited by 136
  19. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  20. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  21. 2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 74
  22. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  23. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 50
  24. 2013Evaluation Measures for Hierarchical Classification: a unified view and novel approachescited by 46
  25. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  26. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 32
  27. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 31
  28. 2022Label Relation Graphs Enhanced Hierarchical Residual Network for Hierarchical Multi-Granularity Classificationcited by 29
  29. 2024Language Models as Hierarchy Encoderscited by 28
  30. 2023Language Model Analysis for Ontology Subsumption Inferencecited by 25
  31. 2023Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Modelscited by 23
  32. 2022Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classificationcited by 21
  33. 2023HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understandingcited by 21
  34. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  35. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 19
  36. 2024Visually Consistent Hierarchical Image Classificationcited by 16
  37. 2024Learning Structured Representations with Hyperbolic Embeddingscited by 16
  38. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilitiescited by 15

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  3. 2025DuoTeach: Dual Role Self-Teaching for Coarse-to-Fine Decision Coordination in Vision--Language Modelscited by 1
  4. 2026Learning Consistent Taxonomic Classification through Hierarchical Reasoningcited by 0
  5. 2026Specificity-aware reinforcement learning for fine-grained open-world classificationcited by 0
  6. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosiscited by 0
  7. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0