mlsanityData to 14 Oct 2026

Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning

arXiv 2309.05904 ↗First posted 12 Sep 2023cs.CVcited by 34

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face crucial challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and the capability of utilizing very limited or even no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a masked contrastive chest X-ray foundation model that tackles these challenges. MaCo explores masked contrastive learning to simultaneously achieve fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It designs a correlation weighting mechanism to adjust the correlation between masked chest X-ray image patches and their corresponding reports, thereby enhancing the model's representation learning capabilities. To evaluate the performance of MaCo, we conducted extensive experiments using 6 well-known open-source X-ray datasets. The experimental results demonstrate the superiority of MaCo over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding. These findings highlight the significant potential of MaCo in advancing a wide range of medical image analysis tasks.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 61 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

30 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimization17 Jul
  2. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias31 May

Other references · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  6. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  7. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  8. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  9. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  10. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  11. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  12. 2020A Survey on Contrastive Self-supervised Learningcited by 554
16 more
  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  2. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  3. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  4. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  5. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  6. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
  7. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  8. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  9. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  10. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  11. 2020Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representationscited by 69
  12. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  13. 2023A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnosticscited by 43
  14. 2023PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysiscited by 40
  15. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  16. 2020Discriminative, Generative and Self-Supervised Approaches for Target-Agnostic Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 14; 2026: 7.

34 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
  3. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  4. 2023Medical Vision Language Pretraining: A surveycited by 24
  5. 2024EchoFM: Foundation Model for Generalizable Echocardiogram Analysiscited by 15
  6. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  7. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  8. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 11
  9. 2024DeViDe: Faceted medical knowledge for improved medical vision-language pre-trainingcited by 10
  10. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  11. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  12. 2025Foundation Models in Medical Imaging: A Review and Outlookcited by 8
22 more
  1. 2025Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translationcited by 8
  2. 2024A multi-modal vision-language model for generalizable annotation-free pathology localizationcited by 7
  3. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  4. 2023Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrievalcited by 6
  5. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  6. 2025DeepSparse: A Foundation Model for Sparse-View CBCT Reconstructioncited by 5
  7. 2025Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Modelscited by 5
  8. 2024Multimodal self-supervised learning for lesion localizationcited by 4
  9. 2024Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniquescited by 3
  10. 2025SimCroP: Radiograph Representation Learning with Similarity-driven Cross-granularity Pre-trainingcited by 3
  11. 2025Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generationcited by 3
  12. 2025Towards Generalist Intelligence in Dentistry: Vision Foundation Models for Oral and Maxillofacial Radiologycited by 3
  13. 2025X-WIN: Building Chest Radiograph World Model via Predictive Sensingcited by 2
  14. 2025Cardiac-CLIP: A Vision-Language Foundation Model for 3D Cardiac CT Imagescited by 1
  15. 2026Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rayscited by 1
  16. 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-trainingcited by 1
  17. 2025BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysiscited by 0
  18. 2025A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysiscited by 0
  19. 2026Representation Learning with Semantic-aware Instance and Sparse Token Alignmentscited by 0
  20. 2026Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modelingcited by 0
  21. 2026AlphaRAD: Grounded Zero-Shot Classification in Chest Radiology via α-Corrected Binary Cross Entropy and Factorized Latent Supervisioncited by 0
  22. 2026PLRS-IC: A Dual-Calibration Framework for Chest X-Ray Vision-Language Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Tasks
  2. 2026GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
  3. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models
  4. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
  5. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
  6. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-ray
  7. 2025Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
  8. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification