# vMFCoOp: Towards Equilibrium on a Unified Hyperspherical Manifold for Prompting Biomedical VLMs

arXiv: https://arxiv.org/abs/2511.09540 · HTML: https://mlsanity.com/paper/2511.09540/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](https://mlsanity.com/paper/2307.06948/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [Lung and Colon Cancer Histopathological Image Dataset (LC25000)](https://mlsanity.com/paper/1912.12142/) (2019)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP](https://mlsanity.com/paper/2404.02285/) (2024)
- [COVID-19 Infection Localization and Severity Grading from Chest X-ray Images](https://mlsanity.com/paper/2103.07985/) (2021)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [Selective Vision-Language Subspace Projection for Few-shot CLIP](https://mlsanity.com/paper/2407.16977/) (2024)
- [Domain Prompt Learning with Quaternion Networks](https://mlsanity.com/paper/2312.08878/) (2023)
- [MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output](https://mlsanity.com/paper/2510.10011/) (2025)
- [Unified Spatial-Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2502.02504/) (2025)
- [HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention](https://mlsanity.com/paper/2402.14185/) (2024)
- [BP-SGCN: Behavioral Pseudo-Label Informed Sparse Graph Convolution Network for Pedestrian and Heterogeneous Trajectory Prediction](https://mlsanity.com/paper/2502.14676/) (2025)
- [Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment](https://mlsanity.com/paper/2305.11439/) (2023)
- [Adaptively Distilled ControlNet: Accelerated Training and Superior Sampling for Medical Image Synthesis](https://mlsanity.com/paper/2507.23652/) (2025)
- [MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention](https://mlsanity.com/paper/2501.03839/) (2025)

## Built on by (most cited first, 3 of 3)

- [RED-Sphere: Hyperspherical Residual Edge Debiasing for Cross-Population Fundus Disease Domain Generalization](https://mlsanity.com/paper/2607.10777/) (2026)
- [Mitigating Class-Tail Undercoverage in Medical Vision-Language Models under Clinical Shift](https://mlsanity.com/paper/2607.28696/) (2026)
- [Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models](https://mlsanity.com/paper/2609.33701/) (2026)

## Abstract (arXiv)

Recent advances in context optimization (CoOp) guided by large language model (LLM)-distilled medical semantic priors offer a scalable alternative to manual prompt engineering and full fine-tuning for adapting biomedical CLIP-based vision-language models (VLMs). However, prompt learning in this context is challenged by semantic misalignment between LLMs and CLIP variants due to divergent training corpora and model architectures; it further lacks scalability across continuously evolving families of foundation models. More critically, pairwise multimodal alignment via conventional Euclidean-space optimization lacks the capacity to model unified representations or apply localized geometric constraints, which tends to amplify modality gaps in complex biomedical imaging and destabilize few-shot adaptation. In this work, we propose vMFCoOp, a framework that inversely estimates von Mises-Fisher (vMF) distributions on a shared Hyperspherical Manifold, aligning semantic biases between arbitrary LLMs and CLIP backbones via Unified Semantic Anchors to achieve robust biomedical prompting and superior few-shot classification. Grounded in three complementary constraints, vMFCoOp demonstrates consistent improvements across 14 medical datasets, 12 medical imaging modalities, and 13 anatomical regions, outperforming state-of-the-art methods in accuracy, generalization, and clinical applicability. This work aims to continuously expand to encompass more downstream applications, and the corresponding resources are intended to be shared through https://github.com/VinyehShaw/UniEqui.