# Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning

arXiv: https://arxiv.org/abs/2309.05904 · HTML: https://mlsanity.com/paper/2309.05904/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 61 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Images](https://mlsanity.com/paper/2302.14042/) (2023)
- [Multimodal Masked Autoencoders Learn Transferable Representations](https://mlsanity.com/paper/2205.14204/) (2022)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [Joint Learning of Localized Representations from Medical Images and Reports](https://mlsanity.com/paper/2112.02889/) (2021)
- [Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports](https://mlsanity.com/paper/2111.03452/) (2021)
- [Advancing Radiograph Representation Learning with Masked Record Modeling](https://mlsanity.com/paper/2301.13155/) (2023)
- [Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representations](https://mlsanity.com/paper/2007.07423/) (2020)
- [Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning](https://mlsanity.com/paper/2102.10680/) (2021)
- [Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias](https://mlsanity.com/paper/2305.19894/) (2023)
- [A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics](https://mlsanity.com/paper/2306.00864/) (2023)
- [M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization](https://mlsanity.com/paper/2307.08347/) (2023)
- [PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis](https://mlsanity.com/paper/2301.00772/) (2023)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [Discriminative, Generative and Self-Supervised Approaches for Target-Agnostic Learning](https://mlsanity.com/paper/2011.06428/) (2020)

## Built on by (most cited first, 30 of 34)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [EchoFM: Foundation Model for Generalizable Echocardiogram Analysis](https://mlsanity.com/paper/2410.23413/) (2024)
- [T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency](https://mlsanity.com/paper/2312.01529/) (2023)
- [G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training](https://mlsanity.com/paper/2312.01522/) (2023)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [DeViDe: Faceted medical knowledge for improved medical vision-language pre-training](https://mlsanity.com/paper/2404.03618/) (2024)
- [Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learning](https://mlsanity.com/paper/2603.07113/) (2026)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translation](https://mlsanity.com/paper/2508.09177/) (2025)
- [A multi-modal vision-language model for generalizable annotation-free pathology localization](https://mlsanity.com/paper/2401.02044/) (2024)
- [Pre-training on High Definition X-ray Images: An Experimental Study](https://mlsanity.com/paper/2404.17926/) (2024)
- [Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval](https://mlsanity.com/paper/2312.15840/) (2023)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [DeepSparse: A Foundation Model for Sparse-View CBCT Reconstruction](https://mlsanity.com/paper/2505.02628/) (2025)
- [Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models](https://mlsanity.com/paper/2506.08990/) (2025)
- [Multimodal self-supervised learning for lesion localization](https://mlsanity.com/paper/2401.01524/) (2024)
- [Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques](https://mlsanity.com/paper/2401.01583/) (2024)
- [SimCroP: Radiograph Representation Learning with Similarity-driven Cross-granularity Pre-training](https://mlsanity.com/paper/2509.08311/) (2025)
- [Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation](https://mlsanity.com/paper/2509.25963/) (2025)
- [Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiology](https://mlsanity.com/paper/2510.14532/) (2025)
- [X-WIN: Building Chest Radiograph World Model via Predictive Sensing](https://mlsanity.com/paper/2511.14918/) (2025)
- [Cardiac-CLIP: A Vision-Language Foundation Model for 3D Cardiac CT Images](https://mlsanity.com/paper/2507.22024/) (2025)
- [Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays](https://mlsanity.com/paper/2603.26049/) (2026)
- [ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training](https://mlsanity.com/paper/2606.00602/) (2026)
- [BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis](https://mlsanity.com/paper/2505.09329/) (2025)
- [A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysis](https://mlsanity.com/paper/2508.13072/) (2025)

## Abstract (arXiv)

Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face crucial challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and the capability of utilizing very limited or even no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a masked contrastive chest X-ray foundation model that tackles these challenges. MaCo explores masked contrastive learning to simultaneously achieve fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model's representation learning capabilities. To evaluate the performance of MaCo, we conducted extensive experiments using 6 well-known open-source X-ray datasets. The experimental results demonstrate the superiority of MaCo over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding. These findings highlight the significant potential of MaCo in advancing a wide range of medical image analysis tasks.