# Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography

arXiv: https://arxiv.org/abs/2405.12255 · HTML: https://mlsanity.com/paper/2405.12255/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 20 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography](https://mlsanity.com/paper/2203.11205/) (2022)
- [CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training](https://mlsanity.com/paper/2310.13292/) (2023)
- [Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays](https://mlsanity.com/paper/2206.12704/) (2022)
- [Anatomy-Driven Pathology Detection on Chest X-rays](https://mlsanity.com/paper/2309.02578/) (2023)
- [ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data](https://mlsanity.com/paper/2308.11194/) (2023)

## Built on by (most cited first, 30 of 38)

- [A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery](https://mlsanity.com/paper/2406.10833/) (2024)
- [Robust image representations with counterfactual contrastive learning](https://mlsanity.com/paper/2409.10365/) (2024)
- [Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.04639/) (2025)
- [Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography](https://mlsanity.com/paper/2409.18119/) (2024)
- [LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers](https://mlsanity.com/paper/2408.07832/) (2024)
- [Bias and Generalizability of Foundation Models across Datasets in Breast Mammography](https://mlsanity.com/paper/2505.10579/) (2025)
- [Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders](https://mlsanity.com/paper/2507.15227/) (2025)
- [Exemplar Med-DETR: Toward Generalized and Robust Lesion Detection in Mammogram Images and beyond](https://mlsanity.com/paper/2507.19621/) (2025)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)
- [Enhancing the Safety of Medical Vision-Language Models by Synthetic Demonstrations](https://mlsanity.com/paper/2506.09067/) (2025)
- [Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammography](https://mlsanity.com/paper/2506.13964/) (2025)
- [Data-Efficient Fine-Tuning of Vision-Language Models for Diagnosis of Alzheimer's Disease](https://mlsanity.com/paper/2509.07613/) (2025)
- [GLAM: Geometry-Guided Local Alignment for Multi-View VLP in Mammography](https://mlsanity.com/paper/2509.10344/) (2025)
- [MammoClean: Toward Reproducible and Bias-Aware AI in Mammography through Dataset Harmonization](https://mlsanity.com/paper/2511.02400/) (2025)
- [MIL-PF: Multiple Instance Learning on Precomputed Features for Mammography Classification](https://mlsanity.com/paper/2603.09374/) (2026)
- [Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms](https://mlsanity.com/paper/2604.19350/) (2026)
- [Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift](https://mlsanity.com/paper/2607.10358/) (2026)
- [Dataset-Origin Signatures and Shortcut Learning in Screening Mammography AI: A Cross-Dataset Case Study](https://mlsanity.com/paper/2607.15416/) (2026)
- [MAMBO: High-Resolution Generative Approach for Mammography Images](https://mlsanity.com/paper/2506.08677/) (2025)
- [Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models](https://mlsanity.com/paper/2509.21102/) (2025)
- [Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Models](https://mlsanity.com/paper/2510.25051/) (2025)
- [MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Prediction](https://mlsanity.com/paper/2510.26151/) (2025)
- [Does medical specialization of VLMs enhance discriminative power?: A comprehensive investigation through feature distribution analysis](https://mlsanity.com/paper/2601.14774/) (2026)
- [Free Lunch in Medical Image Foundation Model Pre-training via Randomized Synthesis and Disentanglement](https://mlsanity.com/paper/2602.12317/) (2026)
- [MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation](https://mlsanity.com/paper/2602.22462/) (2026)
- [OTCHA: Optimal Transport-driven Confidence-aware Latent Hub Alignment for Multi-View Medical Image Classification](https://mlsanity.com/paper/2606.19838/) (2026)
- [MammoFlow: Multiview Mammogram Synthesis with Anatomically Consistent Flow Matching](https://mlsanity.com/paper/2606.28537/) (2026)
- [Boosting Ultrasound Image Classification via Attribute-Guided Dual-Branch Framework](https://mlsanity.com/paper/2607.01648/) (2026)
- [CPR: Chained Perceptual Refinement for Coarse-to-Fine Medical Image Classification](https://mlsanity.com/paper/2607.02591/) (2026)
- [Pitfalls of Administrative Censoring in Survival Models with Time-Indexed Inputs](https://mlsanity.com/paper/2607.10466/) (2026)

## Abstract (arXiv)

The lack of large and diverse training data on Computer-Aided Diagnosis (CAD) in breast cancer detection has been one of the concerns that impedes the adoption of the system. Recently, pre-training with large-scale image text datasets via Vision-Language models (VLM) (eg CLIP) partially addresses the issue of robustness and data efficiency in computer vision (CV). This paper proposes Mammo-CLIP, the first VLM pre-trained on a substantial amount of screening mammogram-report pairs, addressing the challenges of dataset diversity and size. Our experiments on two public datasets demonstrate strong performance in classifying and localizing various mammographic attributes crucial for breast cancer detection, showcasing data efficiency and robustness similar to CLIP in CV. We also propose Mammo-FActOR, a novel feature attribution method, to provide spatial interpretation of representation with sentence-level granularity within mammography reports. Code is available publicly: urlhttps://github.com/batmanlab/Mammo-CLIP.