# Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2501.16282 · HTML: https://mlsanity.com/paper/2501.16282/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models](https://mlsanity.com/paper/2404.00578/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training](https://mlsanity.com/paper/2309.06828/) (2023)
- [MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning](https://mlsanity.com/paper/2312.14574/) (2023)

## Built on by (most cited first, 6 of 6)

- [Core-Periphery Principle Guided State Space Model for Functional Connectome Classification](https://mlsanity.com/paper/2503.14655/) (2025)
- [BrainNet-MoE: Brain-Inspired Mixture-of-Experts Learning for Neurological Disease Identification](https://mlsanity.com/paper/2503.07640/) (2025)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)
- [Domain-Adaptive Diagnosis of Lewy Body Disease with Transferability Aware Transformer](https://mlsanity.com/paper/2507.08839/) (2025)
- [Bridging Brain Connectomes and Clinical Reports for Early Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2508.06565/) (2025)
- [Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offs](https://mlsanity.com/paper/2603.14271/) (2026)

## Abstract (arXiv)

Understanding brain disorders is crucial for accurate clinical diagnosis and treatment. Recent advances in Multimodal Large Language Models (MLLMs) offer a promising approach to interpreting medical images with the support of text descriptions. However, previous research has primarily focused on 2D medical images, leaving richer spatial information of 3D images under-explored, and single-modality-based methods are limited by overlooking the critical clinical information contained in other modalities. To address this issue, this paper proposes Brain-Adapter, a novel approach that incorporates an extra bottleneck layer to learn new knowledge and instill it into the original pre-trained knowledge. The major idea is to incorporate a lightweight bottleneck layer to train fewer parameters while capturing essential information and utilize a Contrastive Language-Image Pre-training (CLIP) strategy to align multimodal data within a unified representation space. Extensive experiments demonstrated the effectiveness of our approach in integrating multimodal data to significantly improve the diagnosis accuracy without high computational costs, highlighting the potential to enhance real-world diagnostic workflows.