# Boosting Medical Visual Understanding From Multi-Granular Language Learning

arXiv: https://arxiv.org/abs/2511.15943 · HTML: https://mlsanity.com/paper/2511.15943/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 105 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 30 of 58)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [Multi-label Image Recognition by Recurrently Discovering Attentional Regions](https://mlsanity.com/paper/1711.02816/) (2017)
- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)

## Built on by (most cited first, 2 of 2)

- [PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation](https://mlsanity.com/paper/2602.23297/) (2026)
- [OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis](https://mlsanity.com/paper/2607.25108/) (2026)

## Abstract (arXiv)

Recent advances in image-text pretraining have significantly enhanced visual understanding by aligning visual and textual representations. Contrastive Language-Image Pretraining (CLIP) has played a pivotal role in multimodal learning. However, its focus on single-label, single-granularity alignment limits its effectiveness in complex domains such as medical imaging, where images often correspond to multiple high-level labels (e.g., disease categories) across different annotation granularities (e.g., diagnostic description, clinical explanation). To address this, we propose Multi-Granular Language Learning (MGLL), a contrastive learning framework designed to improve both multi-label and cross-granularity alignment. MGLL leverages structured multi-label supervision, integrates textual descriptions across granularities, and introduces soft-label supervision with point-wise constraints to enhance alignment. MGLL employs smooth Kullback-Leibler (KL) divergence to ensure cross-granularity consistency while maintaining computational efficiency as a plug-and-play module for vision-language models. Pretrained on our constructed large-scale multi-granular datasets and evaluated across multiple datasets, MGLL outperforms other state-of-the-art methods in downstream tasks. The code is available at https://github.com/HUANGLIZI/MGLL.