# Task-driven Prompt Evolution for Foundation Models

arXiv: https://arxiv.org/abs/2310.17128 · HTML: https://mlsanity.com/paper/2310.17128/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 8 of 8)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [SAM on Medical Images: A Comprehensive Study on Three Prompt Modes](https://mlsanity.com/paper/2305.00035/) (2023)
- [Semantic Segmentation using Adversarial Networks](https://mlsanity.com/paper/1611.08408/) (2016)
- [Domain Generalization on Medical Imaging Classification using Episodic Training with Task Augmentation](https://mlsanity.com/paper/2106.06908/) (2021)

## Built on by (most cited first, 1 of 1)

- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)

## Abstract (arXiv)

Promptable foundation models, particularly Segment Anything Model (SAM), have emerged as a promising alternative to the traditional task-specific supervised learning for image segmentation. However, many evaluation studies have found that their performance on medical imaging modalities to be underwhelming compared to conventional deep learning methods. In the world of large pre-trained language and vision-language models, learning prompt from downstream tasks has achieved considerable success in improving performance. In this work, we propose a plug-and-play Prompt Optimization Technique for foundation models like SAM (SAMPOT) that utilizes the downstream segmentation task to optimize the human-provided prompt to obtain improved performance. We demonstrate the utility of SAMPOT on lung segmentation in chest X-ray images and obtain an improvement on a significant number of cases (~75%) over human-provided initial prompts. We hope this work will lead to further investigations in the nascent field of automatic visual prompt-tuning.