mlsanityData to 7 Oct 2026

K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging

arXiv 2603.06340 ↗First posted 6 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large-scale biomedical vision-language models (VLMs) adapted on high-end imaging (e.g., CT) often fail to transfer to frontline low-end modalities (e.g., radiography), collapsing into modality-specific shortcuts. We propose K-MaT (Knowledge-Anchored Manifold Transport), a prompt-learning framework that transfers decision structures to low-end modalities without requiring low-end training images. K-MaT factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the visually-grounded high-end space using Fused Gromov-Wasserstein optimal transport. We evaluate K-MaT on four cross-modal benchmarks, including dermoscopy, mammography to ultrasound, and CT to chest X-ray. K-MaT achieves state-of-the-art results, improving the average harmonic mean of accuracy to 44.1% (from BiomedCoOp's 42.0%) and macro-F1 to 36.2%. Notably, on the challenging breast imaging task, it mitigates the catastrophic forgetting seen in standard methods like CoOp (which drops to 27.0% accuracy on the low-end), preserving robust performance across modalities. Aligning prompt manifolds via optimal transport provides a highly effective route for the zero-shot cross-modal deployment of medical VLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 17 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

10 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models21 Nov

Other references · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  4. 2020In Search of Lost Domain Generalizationcited by 1,042
  5. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  6. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  7. 2018Optimal Transport for structured data with application on graphscited by 170
  8. 2023Domain Generalization for Medical Image Analysis: A Reviewcited by 47
  9. 2024A Systematic Review of Generalization Research in Medical Image Classificationcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study
  2. 2025vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs
  3. 2025Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks
  4. 2025Anatomy-Grounded Weakly Supervised Prompt Tuning for Chest X-ray Latent Diffusion Models
  5. 2025Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models
  6. 2025Medical Knowledge Intervention Prompt Tuning for Medical Image Classification
  7. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation
  8. 2025VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine