mlsanityData to 14 Oct 2026

WISE: Weak-Supervision-Guided Step-by-Step Explanations for Multimodal LLMs in Image Classification

arXiv 2509.17740 ↗First posted 22 Sep 2025cs.CV cs.CLcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have shown promise in visual-textual reasoning, with Multimodal Chain-of-Thought (MCoT) prompting significantly enhancing interpretability. However, existing MCoT methods rely on rationale-rich datasets and largely focus on inter-object reasoning, overlooking the intra-object understanding crucial for image classification. To address this gap, we propose WISE, a Weak-supervision-guided Step-by-step Explanation method that augments any image classification dataset with MCoTs by reformulating the concept-based representations from Concept Bottleneck Models (CBMs) into concise, interpretable reasoning chains under weak supervision. Experiments across ten datasets show that our generated MCoTs not only improve interpretability by 37% but also lead to gains in classification accuracy when used to fine-tune MLLMs. Our work bridges concept-based interpretability and generative MCoT reasoning, providing a generalizable framework for enhancing MLLMs in fine-grained visual understanding.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 42 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

23 of 42 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination15 Nov
  2. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLM24 Apr
  3. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning12 Jan

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  5. 2020Concept Bottleneck Modelscited by 1,032
  6. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  9. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  10. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  11. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  12. 2019Interpretability Beyond Classification Output: Semantic Bottleneck Networkscited by 49
8 more
  1. 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatologycited by 28
  2. 2023Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decisioncited by 27
  3. 2018A Large-scale Attribute Dataset for Zero-shot Learningcited by 21
  4. 2024Integrating Clinical Knowledge into Concept Bottleneck Modelscited by 17
  5. 2024PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Drivingcited by 17
  6. 2025MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessmentcited by 12
  7. 2025Enhancing Interpretable Image Classification Through LLM Agents and Conditional Concept Bottleneck Modelscited by 6
  8. 2025Interpretable Few-Shot Retinal Disease Diagnosis with Concept-Guided Prompting of Vision-Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognitioncited by 1
  2. 2025ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inferencecited by 0
  3. 2026NeRD: Neuro-Symbolic Rule Distillation for Efficient Ontology-Grounded Chain-of-Thought in Medical Image Diagnosiscited by 0
  4. 2026VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  2. 2026Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers
  3. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
  4. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities
  5. 2025On Large Multimodal Models as Open-World Image Classifiers
  6. 2025CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification
  7. 2024A Concept-Based Explainability Framework for Large Multimodal Models
  8. 2026CBX-Bench: A Human-Aligned MLLM Council for Benchmarking Concept Bottleneck Model Explanations