# From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology

arXiv: https://arxiv.org/abs/2511.11984 · HTML: https://mlsanity.com/paper/2511.11984/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis](https://mlsanity.com/paper/2108.05930/) (2021)
- [A Systematic Review of Few-Shot Learning in Medical Imaging](https://mlsanity.com/paper/2309.11433/) (2023)
- [Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomics, and Demographic Data](https://mlsanity.com/paper/2203.04419/) (2022)
- [Foundation Models in Computational Pathology: A Review of Challenges, Opportunities, and Impact](https://mlsanity.com/paper/2502.08333/) (2025)
- [FHIST: A Benchmark for Few-shot Classification of Histological Images](https://mlsanity.com/paper/2206.00092/) (2022)
- [A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Tasks](https://mlsanity.com/paper/2501.15724/) (2025)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Models](https://mlsanity.com/paper/2408.14496/) (2024)
- [Style Transfer: From Stitching to Neural Networks](https://mlsanity.com/paper/2409.00606/) (2024)
- [Pathology Foundation Models](https://mlsanity.com/paper/2407.21317/) (2024)
- [Glo-In-One-v2: Holistic Identification of Glomerular Cells, Tissues, and Lesions in Human and Mouse Histopathology](https://mlsanity.com/paper/2411.16961/) (2024)
- [Benchmarking PathCLIP for Pathology Image Analysis](https://mlsanity.com/paper/2401.02651/) (2024)
- [Comparative Analysis of Different Efficient Fine Tuning Methods of Large Language Models (LLMs) in Low-Resource Setting](https://mlsanity.com/paper/2405.13181/) (2024)
- [Effects of annotation granularity in deep learning models for histopathological images](https://mlsanity.com/paper/2001.04663/) (2020)
- [Fine-grained Multi-class Nuclei Segmentation with Molecular-empowered All-in-SAM Model](https://mlsanity.com/paper/2508.15751/) (2025)
- [Glo-VLMs: Leveraging Vision-Language Models for Fine-Grained Diseased Glomerulus Classification](https://mlsanity.com/paper/2508.15960/) (2025)
- [Anomaly Detection and Early Warning Mechanism for Intelligent Monitoring Systems in Multi-Cloud Environments Based on LLM](https://mlsanity.com/paper/2506.07407/) (2025)

## Built on by (most cited first, 1 of 1)

- [Seeing Abnormal from Normal: Glomerular Abnormality in Representations of Normal Renal Morphology](https://mlsanity.com/paper/2609.19444/) (2026)

## Abstract (arXiv)

Fine-grained glomerular subtyping is essential for kidney biopsy interpretation, yet it presents a harder setting than the broad organ- or disease-level recognition tasks commonly used in vision-language model (VLM) evaluation. In this task, all categories arise within the same renal microanatomic unit, and clinically meaningful subtypes often differ by subtle, overlapping, and continuum-like morphological cues. Clinically valuable labels are also scarce, making fully supervised image-only paradigms difficult to translate to practice. Although VLMs are promising for data-efficient pathology learning, it remains unclear how their cross-modal representations should be adapted and evaluated for such within-structure discrimination under realistic few-shot constraints. To our knowledge, this study establishes the first fine-grained renal pathology benchmark for few-shot VLM adaptation and systematically examines how supervision level, model design, and adaptation strategy shape both classification performance and multimodal representation structure. Beyond accuracy, AUC, and F1, we assess image-text alignment, positive-negative similarity separation, embedding geometry, and calibration (ECE), asking when performance gains reflect more reliable cross-modal understanding. Across renal pathology cohorts, additional supervision consistently improves discrimination, but reliable adaptation is not explained by alignment alone; stronger performance is associated with larger class separation, more coherent embedding organization, and better awareness of confidence-related failure modes under domain variation. Within this retrospective two-cohort evaluation, these findings move assessment beyond coarse classification and accuracy alone and provide comparative evidence to inform model selection and future translational validation. Code: https://github.com/ddrrnn123/glo-vlms