# Improvise, Adapt, Overcome -- Telescopic Adapters for Efficient Fine-tuning of Vision Language Models in Medical Imaging

arXiv: https://arxiv.org/abs/2512.13855 · HTML: https://mlsanity.com/paper/2512.13855/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2306.07967/) (2023)
- [Zero-shot Referring Image Segmentation with Global-Local Context Features](https://mlsanity.com/paper/2303.17811/) (2023)
- [Less is More: Pay Less Attention in Vision Transformers](https://mlsanity.com/paper/2105.14217/) (2021)
- [On Pre-Trained Image Features and Synthetic Images for Deep Learning](https://mlsanity.com/paper/1710.10710/) (2017)
- [NeoUNet: Towards accurate colon polyp segmentation and neoplasm detection](https://mlsanity.com/paper/2107.05023/) (2021)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Deep Network Approximation: Beyond ReLU to Diverse Activation Functions](https://mlsanity.com/paper/2307.06555/) (2023)
- [A Survey of Unsupervised Domain Adaptation for Visual Recognition](https://mlsanity.com/paper/2112.06745/) (2021)
- [Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models](https://mlsanity.com/paper/2308.07706/) (2023)

## Built on by (most cited first, 2 of 2)

- [BackFlush: Knowledge-Free Backdoor Detection and Elimination with Watermark Preservation in Large Language Models](https://mlsanity.com/paper/2605.12529/) (2026)
- [Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement](https://mlsanity.com/paper/2609.34528/) (2026)

## Abstract (arXiv)

Adapting Vision Language Segmentation Models (VLSMs) to medical imaging domains requires significant computational overhead when using conventional fine-tuning approaches. Existing Parameter-Efficient Fine-Tuning (PEFT) methods apply uniform adapter dimensions across all transformer layers, leading to suboptimal parameter allocation and reduced adaptation efficiency. We introduce Telescopic Adapters, a novel PEFT framework that employs depth-aware scaling to progressively increase adapter capacity from shallow to deep transformer layers. Our method integrates lightweight bottleneck modules within CLIPSeg's vision and text encoders, with adapter dimensions dynamically scaled based on layer depth and semantic relevance. Using only 613k trainable parameters--244x fewer than end-to-end fine-tuning, Telescopic Adapters achieve superior performance across five diverse medical datasets spanning polyp segmentation, skin lesion detection, and breast ultrasound imaging. Comprehensive ablation studies demonstrate that deeper layers require substantially more adaptation capacity than shallow layers, validating our telescopic scaling hypothesis. Our approach establishes a new paradigm for efficient medical VLSM fine-tuning, enabling deployment in resource-constrained clinical environments while maintaining competitive segmentation accuracy. Our source code is publicly available at https://github.com/Ujjwal238/Telescopic_adapters