mlsanityData to 14 Oct 2026

Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs

arXiv 2509.09732 ↗First posted 10 Sep 2025cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree-based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 27 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

18 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Improve Vision Language Model Chain-of-thought Reasoning21 Oct

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  6. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  7. 2025Vision-Language Models Do Not Understand Negationcited by 72
  8. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  9. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  10. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  11. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14
  12. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 13
5 more
  1. 2024How Well Can Vision Language Models See Image Details?cited by 10
  2. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
  3. 2024Tree of Attributes Prompt Learning for Vision-Language Modelscited by 7
  4. 2024Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Modelscited by 4
  5. 2024In the Era of Prompt Learning with Vision-Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhance Reasoning Ability of Visual-Language Models via Large Language Models
  2. 2024Rethinking VLMs and LLMs for Image Classification
  3. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologies
  4. 2026Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?
  5. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
  6. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  7. 2026Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models
  8. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs