# MedPrompt: Cross-Modal Prompting for Multi-Task Medical Image Translation

arXiv: https://arxiv.org/abs/2310.02663 · HTML: https://mlsanity.com/paper/2310.02663/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [Few-Shot Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1905.01723/) (2019)
- [U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation](https://mlsanity.com/paper/1907.10830/) (2019)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Medical Image Synthesis with Context-Aware Generative Adversarial Networks](https://mlsanity.com/paper/1612.05362/) (2016)
- [Visual Prompt Tuning for Generative Transfer Learning](https://mlsanity.com/paper/2210.00990/) (2022)
- [High-Resolution Photorealistic Image Translation in Real-Time: A Laplacian Pyramid Translation Network](https://mlsanity.com/paper/2105.09188/) (2021)
- [SynthRAD2023 Grand Challenge dataset: generating synthetic CT for radiotherapy](https://mlsanity.com/paper/2303.16320/) (2023)
- [Bidirectional Mapping Generative Adversarial Networks for Brain MR to PET Synthesis](https://mlsanity.com/paper/2008.03483/) (2020)
- [Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain](https://mlsanity.com/paper/2011.04145/) (2020)
- [3D Brain Reconstruction by Hierarchical Shape-Perception Network from a Single Incomplete Image](https://mlsanity.com/paper/2107.11010/) (2021)
- [A Prior Guided Adversarial Representation Learning and Hypergraph Perceptual Network for Predicting Abnormal Connections of Alzheimer's Disease](https://mlsanity.com/paper/2110.09302/) (2021)

## Built on by (most cited first, 13 of 13)

- [2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification](https://mlsanity.com/paper/2412.00678/) (2024)
- [SMAFormer: Synergistic Multi-Attention Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2409.00346/) (2024)
- [Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation](https://mlsanity.com/paper/2401.16923/) (2024)
- [Test-Time Intensity Consistency Adaptation for Shadow Detection](https://mlsanity.com/paper/2410.07695/) (2024)
- [UWFormer: Underwater Image Enhancement via a Semi-Supervised Multi-Scale Transformer](https://mlsanity.com/paper/2310.20210/) (2023)
- [Implicit Multi-Spectral Transformer: An Lightweight and Effective Visible to Infrared Image Translation Model](https://mlsanity.com/paper/2404.07072/) (2024)
- [LensNet: An End-to-End Learning Framework for Empirical Point Spread Function Modeling and Lensless Imaging Reconstruction](https://mlsanity.com/paper/2505.01755/) (2025)
- [MCP-MedSAM: A Powerful Lightweight Medical Segment Anything Model Trained with a Single GPU in Just One Day](https://mlsanity.com/paper/2412.05888/) (2024)
- [IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities](https://mlsanity.com/paper/2410.18551/) (2024)
- [DLEN: Dual Branch of Transformer for Low-Light Image Enhancement in Dual Domains](https://mlsanity.com/paper/2501.12235/) (2025)
- [Medical Visual Prompting (MVP): A Unified Framework for Versatile and High-Quality Medical Image Segmentation](https://mlsanity.com/paper/2404.01127/) (2024)
- [Adaptive Query Prompting for Multi-Domain Landmark Detection](https://mlsanity.com/paper/2404.01194/) (2024)
- [Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pests](https://mlsanity.com/paper/2510.00547/) (2025)

## Abstract (arXiv)

Cross-modal medical image translation is an essential task for synthesizing missing modality data for clinical diagnosis. However, current learning-based techniques have limitations in capturing cross-modal and global features, restricting their suitability to specific pairs of modalities. This lack of versatility undermines their practical usefulness, particularly considering that the missing modality may vary for different cases. In this study, we present MedPrompt, a multi-task framework that efficiently translates different modalities. Specifically, we propose the Self-adaptive Prompt Block, which dynamically guides the translation network towards distinct modalities. Within this framework, we introduce the Prompt Extraction Block and the Prompt Fusion Block to efficiently encode the cross-modal prompt. To enhance the extraction of global features across diverse modalities, we incorporate the Transformer model. Extensive experimental results involving five datasets and four pairs of modalities demonstrate that our proposed model achieves state-of-the-art visual quality and exhibits excellent generalization capability.