mlsanityData to 14 Oct 2026

Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology

arXiv 2609.17868 ↗First posted 15 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Pathology vision-language models are commonly built by pretraining or fine-tuning large encoders on paired image-caption data. We asked whether a pathology vision-language model can instead be assembled by parameter-efficient alignment of frozen unimodal foundation models, leaving their pretrained representations untouched. Here we present Lumen, which aligns frozen Virchow2 and BioMedBERT backbones using rank-4 adapters and projection heads, training only 0.40% of the total parameters on the public QUILT-1M corpus. Across nine public zero-shot patch benchmarks, Lumen achieved the highest mean chance-corrected balanced accuracy, 0.546 versus 0.461 for the strongest baseline (paired difference 0.086, 95% CI 0.042-0.136). On lymph-node metastasis detection, Lumen reached an AUROC of 0.964 (95% CI 0.956-0.971) on 4,214 held-out internal slides and 0.955 (95% CI 0.942-0.966) on 2,368 slides across nine external cohorts and six organs. At the internally calibrated threshold, it outperformed all vision-language baselines, with a balanced accuracy of 0.909 (95% CI 0.896-0.923) internally and 0.915 (95% CI 0.902-0.929) externally. Lumen performed competitively across the evaluations, with the exception of cross-modal retrieval, where it ranked third behind CONCH and PathGen-L/14. Fully fine-tuning both encoders gave Lumen no consistent benefit over low-rank adaptation, although it improved retrieval. Aligning frozen unimodal foundation models therefore yields strong and transferable performance at patch and slide level while training only a small fraction of the parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 40 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

22 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosis17 Dec
  2. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration28 Jun
  3. 2023PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology24 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  4. 2021Image Segmentation Using Text and Image Promptscited by 512
  5. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  6. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 194
  7. 2024Multimodal Whole Slide Foundation Model for Pathologycited by 162
  8. 2024Virchow2: Scaling Self-Supervised Mixed Magnification Models in Pathologycited by 161
  9. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  10. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
  11. 2023Artificial Intelligence for Digital and Computational Pathologycited by 108
  12. 2023Improving Multimodal Datasets with Image Captioningcited by 93
7 more
  1. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  2. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 56
  3. 2024Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgettingcited by 17
  4. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  5. 2025HISTAI: An Open-Source, Large-Scale Whole Slide Image Dataset for Computational Pathologycited by 14
  6. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  7. 2025Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology
  2. 2025Democratising Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling
  3. 2026FFM-CP: Cross-Backbone Fusion of Vision-Language Foundation Models for Few-Shot Computational Pathology
  4. 2026HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
  5. 2024CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment
  6. 2026LaGuadia: Language-Guided Adaptive Distillation from Pathology Foundation Models
  7. 2026Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation
  8. 2025Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design