mlsanityData to 7 Oct 2026

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

arXiv 2606.04613 ↗First posted 3 Jun 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited. Existing representation alignment metrics are symmetric, collapsing both modalities into a single score and hiding which modality drives cross-modal degradation. We introduce the Spectral Alignment Score (SAS), an asymmetric metric that projects both modalities onto the principal eigenbasis of an anchor modality and computes eigenvalue-weighted per-eigenmode correlations, resulting in directional scores whose difference quantifies modality information imbalance. We embed SAS within a benchmarking framework evaluating 15 VLMs across natural and medical image-text datasets alongside 6 alignment metrics and bidirectional retrieval. Our experiments show that medical images retain richer structural information than their paired clinical reports, a directional asymmetry invisible to all competing metrics, and that SAS achieves the strongest zero-label correlation with retrieval performance in the medical domain, positioning it as a practical diagnostic tool for clinical deployment. Code is available at this URL: https://github.com/iamalegambetti/medical-vlms-assessment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 4 of its 29 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

4 of 29 references.

References · 4

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  3. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  4. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MedLayBench-V: A Large-Scale Benchmark for Expert-Lay Semantic Alignment in Medical Vision Language Models
  2. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis
  3. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review
  4. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
  5. 2026MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Images
  6. 2025How Far Have Medical Vision-Language Models Come? A Comprehensive Benchmarking Study
  7. 2026Medical Context Distorts Decisions in Clinical Vision Language Models
  8. 2026CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis