mlsanityData to 14 Oct 2026

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv 2512.21414 ↗First posted 24 Dec 2025cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools. Broadly, these frameworks leverage VLMs and a pre-specified toolbox to decompose the prediction task into multiple tool calls (often deep learning models) which are composed to make a prediction. The dominant approach to composing tools is using text, via function calls embedded in VLM-generated code or natural language. However, these methods often perform poorly on medical image understanding, where salient information is encoded as spatially-localized features that are difficult to compose or fuse via text alone. To address this, we propose a tool-use framework for medical image understanding called the Tool Bottleneck Framework (TBF), which composes VLM-selected tools using a learned Tool Bottleneck Model (TBM). For a given image and task, TBF leverages an off-the-shelf medical VLM to select tools from a toolbox that each extract clinically-relevant features. Instead of text-based composition, these tools are composed by the TBM, which computes and fuses the tool outputs using a neural network before outputting the final prediction. We propose a simple and effective strategy for TBMs to make predictions with any arbitrary VLM tool selection. Overall, our framework not only improves tool-use in medical imaging contexts, but also yields more interpretable, clinically-grounded predictors. We evaluate TBF on tasks in histopathology and dermatology and find that these advantages enable our framework to perform on par with or better than deep learning-based classifiers, VLMs, and state-of-the-art tool-use frameworks, with particular gains in data-limited regimes. The project details and the code are available at https://christinaliu2020.github.io/tbm/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 54 references that are in the map. First posted on arXiv 24 Dec 2025.

Builds on

40 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2014Going Deeper with Convolutionscited by 4,241
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
27 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  2. 2020Concept Bottleneck Modelscited by 1,032
  3. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  4. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  5. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  6. 2017Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)cited by 549
  7. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  8. 2025MedGemma Technical Reportcited by 436
  9. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 388
  10. 2020Rethinking Pre-training and Self-trainingcited by 352
  11. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  12. 2023Label-Free Concept Bottleneck Modelscited by 285
  13. 2023Probabilistic Concept Bottleneck Modelscited by 101
  14. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  15. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  16. 2018Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Imagescited by 42
  17. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  18. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  19. 2023Concept Bottleneck with Visual Concept Filtering for Explainable Medical Image Classificationcited by 21
  20. 2025RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thoughtcited by 19
  21. 2024Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classificationcited by 18
  22. 2024Integrating Clinical Knowledge into Concept Bottleneck Modelscited by 17
  23. 2023Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitiscited by 11
  24. 2024What Makes a Maze Look Like a Maze?cited by 11
  25. 2017Fully Convolutional Neural Networks to Detect Clinical Dermoscopic Featurescited by 9
  26. 2025Graph Concept Bottleneck Modelscited by 9
  27. 2024Knockout: A simple way to handle missing inputscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification
  2. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  3. 2025Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation
  4. 2026Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
  5. 2026MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
  6. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
  7. 2026Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning
  8. 2026Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models