# Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

arXiv: https://arxiv.org/abs/2401.01583 · HTML: https://mlsanity.com/paper/2401.01583/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports](https://mlsanity.com/paper/2111.03452/) (2021)
- [Annotation-efficient deep learning for automatic medical image segmentation](https://mlsanity.com/paper/2012.04885/) (2020)
- [Advancing Radiograph Representation Learning with Masked Record Modeling](https://mlsanity.com/paper/2301.13155/) (2023)
- [Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias](https://mlsanity.com/paper/2305.19894/) (2023)
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](https://mlsanity.com/paper/2306.00864/) (2023)
- [M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization](https://mlsanity.com/paper/2307.08347/) (2023)
- [PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis](https://mlsanity.com/paper/2301.00772/) (2023)
- [Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning](https://mlsanity.com/paper/2309.05904/) (2023)
- [D-UNet: a dimension-fusion U shape network for chronic stroke lesion segmentation](https://mlsanity.com/paper/1908.05104/) (2019)

## Built on by (most cited first, 3 of 3)

- [Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinement](https://mlsanity.com/paper/2401.11421/) (2024)
- [MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting](https://mlsanity.com/paper/2509.03800/) (2025)
- [Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models](https://mlsanity.com/paper/2504.15929/) (2025)

## Abstract (arXiv)

The development of medical vision-language foundation models has attracted significant attention in the field of medicine and healthcare due to their promising prospect in various clinical applications. While previous studies have commonly focused on feature learning at a single learning scale, investigation on integrating multi-scale information is lacking, which may hinder the potential for mutual reinforcement among these features. This paper aims to bridge this gap by proposing a method that effectively exploits multi-scale information to enhance the performance of medical foundation models. The proposed method simultaneously exploits features at the local, instance, modality and global aspects, facilitating comprehensive representation learning within the models. We evaluate the effectiveness of the proposed method on six open-source datasets across different clinical tasks, demonstrating its ability to enhance the performance of medical foundation models.