# AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis

arXiv: https://arxiv.org/abs/2408.02001 · HTML: https://mlsanity.com/paper/2408.02001/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 4 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification](https://mlsanity.com/paper/1903.03313/) (2019)
- [Sparse Linear Concept Discovery Models](https://mlsanity.com/paper/2308.10782/) (2023)
- [The Doctor Just Won't Accept That!](https://mlsanity.com/paper/1711.08037/) (2017)

## Built on by (most cited first, 12 of 12)

- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions](https://mlsanity.com/paper/2507.23784/) (2025)
- [IP-CRR: Information Pursuit for Interpretable Classification of Chest Radiology Reports](https://mlsanity.com/paper/2505.00191/) (2025)
- [Matryoshka Concept Bottleneck Models](https://mlsanity.com/paper/2605.20612/) (2026)
- [Training-free Test-time Improvement for Explainable Medical Image Classification](https://mlsanity.com/paper/2506.18070/) (2025)
- [XDR-LVLM: An Explainable Vision-Language Large Model for Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2508.15168/) (2025)
- [Automated Genomic Interpretation via Concept Bottleneck Models for Medical Robotics](https://mlsanity.com/paper/2510.01618/) (2025)
- [Parameter-Efficient Construction of the Rashomon Slice for Concept Bottleneck Models](https://mlsanity.com/paper/2511.19636/) (2025)
- [Learning Label-Efficient Interpretable Medical Image Diagnosis via Semi-supervised Hypergraph Concept Bottleneck Model](https://mlsanity.com/paper/2606.01698/) (2026)
- [Enhancing Performance of Explainable AI Models with Constrained Concept Refinement](https://mlsanity.com/paper/2502.06775/) (2025)
- [Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning](https://mlsanity.com/paper/2603.08921/) (2026)
- [Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models](https://mlsanity.com/paper/2606.04326/) (2026)

## Abstract (arXiv)

The integration of vision-language models such as CLIP and Concept Bottleneck Models (CBMs) offers a promising approach to explaining deep neural network (DNN) decisions using concepts understandable by humans, addressing the black-box concern of DNNs. While CLIP provides both explainability and zero-shot classification capability, its pre-training on generic image and text data may limit its classification accuracy and applicability to medical image diagnostic tasks, creating a transfer learning problem. To maintain explainability and address transfer learning needs, CBM methods commonly design post-processing modules after the bottleneck module. However, this way has been ineffective. This paper takes an unconventional approach by re-examining the CBM framework through the lens of its geometrical representation as a simple linear classification system. The analysis uncovers that post-CBM fine-tuning modules merely rescale and shift the classification outcome of the system, failing to fully leverage the system's learning potential. We introduce an adaptive module strategically positioned between CLIP and CBM to bridge the gap between source and downstream domains. This simple yet effective approach enhances classification performance while preserving the explainability afforded by the framework. Our work offers a comprehensive solution that encompasses the entire process, from concept discovery to model training, providing a holistic recipe for leveraging the strengths of GPT, CLIP, and CBM.