# BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning

arXiv: https://arxiv.org/abs/2408.07440 · HTML: https://mlsanity.com/paper/2408.07440/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 20 of 20)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Backdoor Learning: A Survey](https://mlsanity.com/paper/2007.08745/) (2020)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [Poisoning and Backdooring Contrastive Learning](https://mlsanity.com/paper/2106.09667/) (2021)
- [On the Adversarial Robustness of Multi-Modal Foundation Models](https://mlsanity.com/paper/2308.10741/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [FIBA: Frequency-Injection based Backdoor Attack in Medical Image Analysis](https://mlsanity.com/paper/2112.01148/) (2021)
- [Text-guided Foundation Model Adaptation for Pathological Image Classification](https://mlsanity.com/paper/2307.14901/) (2023)
- [Learning to Prompt with Text Only Supervision for Vision-Language Models](https://mlsanity.com/paper/2401.02418/) (2024)
- [Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models](https://mlsanity.com/paper/2401.12215/) (2024)
- [Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning](https://mlsanity.com/paper/2303.12214/) (2023)
- [Backdoor Attack and Defense in Federated Generative Adversarial Network-based Medical Image Synthesis](https://mlsanity.com/paper/2210.10886/) (2022)
- [Explainability Matters: Backdoor Attacks on Medical Imaging](https://mlsanity.com/paper/2101.00008/) (2020)

## Built on by (most cited first, 16 of 16)

- [Backdooring Vision-Language Models with Out-Of-Distribution Data](https://mlsanity.com/paper/2410.01264/) (2024)
- [O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models](https://mlsanity.com/paper/2503.12096/) (2025)
- [On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?](https://mlsanity.com/paper/2505.15425/) (2025)
- [Noise is an Efficient Learner for Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2502.06019/) (2025)
- [Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on Breast Ultrasound Images](https://mlsanity.com/paper/2412.09910/) (2024)
- [Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models](https://mlsanity.com/paper/2605.08787/) (2026)
- [Calibration-Aware Prompt Learning for Medical Vision-Language Models](https://mlsanity.com/paper/2509.15226/) (2025)
- [A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models](https://mlsanity.com/paper/2510.26441/) (2025)
- [SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning](https://mlsanity.com/paper/2506.22506/) (2025)
- [Backdoor Unlearning by Linear Task Decomposition](https://mlsanity.com/paper/2510.14845/) (2025)
- [Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs](https://mlsanity.com/paper/2607.16726/) (2026)
- [StealthMark: Harmless and Stealthy Ownership Verification for Medical Segmentation via Uncertainty-Guided Backdoors](https://mlsanity.com/paper/2601.17107/) (2026)
- [BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs](https://mlsanity.com/paper/2604.17629/) (2026)
- [ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models](https://mlsanity.com/paper/2606.31587/) (2026)
- [Backdoor Learning in Language Models and Vision-Language Models](https://mlsanity.com/paper/2608.18095/) (2026)
- [Harnessing SPAD Noise: Test-Time Gated Attention for Indoor Depth Estimation](https://mlsanity.com/paper/2610.15431/) (2026)

## Abstract (arXiv)

Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. Code is available at https://asif-hanif.github.io/baple/.