# Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training

arXiv: https://arxiv.org/abs/2307.07246 · HTML: https://mlsanity.com/paper/2307.07246/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis](https://mlsanity.com/paper/1908.06912/) (2019)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [NegBio: a high-performance tool for negation and uncertainty detection in radiology reports](https://mlsanity.com/paper/1712.05898/) (2017)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training](https://mlsanity.com/paper/2303.00874/) (2023)
- [RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting](https://mlsanity.com/paper/2107.02104/) (2021)
- [Breaking with Fixed Set Pathology Recognition through Report-Guided Contrastive Training](https://mlsanity.com/paper/2205.07139/) (2022)
- [Rethinking Graph Convolutional Networks in Knowledge Graph Completion](https://mlsanity.com/paper/2202.05679/) (2022)
- [Weakly-Supervised Segmentation for Disease Localization in Chest X-Ray Images](https://mlsanity.com/paper/2007.00748/) (2020)

## Built on by (most cited first, 13 of 13)

- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation](https://mlsanity.com/paper/2312.08078/) (2023)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval](https://mlsanity.com/paper/2312.15840/) (2023)
- [Enhancing medical vision-language contrastive learning via inter-matching relation modelling](https://mlsanity.com/paper/2401.10501/) (2024)
- [ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Tasks](https://mlsanity.com/paper/2402.17298/) (2024)
- [Position-Guided Prompt Learning for Anomaly Detection in Chest X-Rays](https://mlsanity.com/paper/2405.11976/) (2024)
- [SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI](https://mlsanity.com/paper/2503.19801/) (2025)
- [Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Records](https://mlsanity.com/paper/2505.09435/) (2025)
- [Foundation Models in Biomedical Imaging: Turning Hype into Reality](https://mlsanity.com/paper/2512.15808/) (2025)
- [Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering](https://mlsanity.com/paper/2510.08791/) (2025)
- [VLCD: Vision-Language Contrastive Distillation for Accurate and Efficient Automatic Placenta Analysis](https://mlsanity.com/paper/2506.02229/) (2025)
- [MoME: Mixture of Visual Language Medical Experts for Medical Imaging Segmentation](https://mlsanity.com/paper/2510.26996/) (2025)

## Abstract (arXiv)

The foundation models based on pre-training technology have significantly advanced artificial intelligence from theoretical to practical applications. These models have facilitated the feasibility of computer-aided diagnosis for widespread use. Medical contrastive vision-language pre-training, which does not require human annotations, is an effective approach for guiding representation learning using description information in diagnostic reports. However, the effectiveness of pre-training is limited by the large-scale semantic overlap and shifting problems in medical field. To address these issues, we propose the Knowledge-Boosting Contrastive Vision-Language Pre-training framework (KoBo), which integrates clinical knowledge into the learning of vision-language semantic consistency. The framework uses an unbiased, open-set sample-wise knowledge representation to measure negative sample noise and supplement the correspondence between vision-language mutual information and clinical knowledge. Extensive experiments validate the effect of our framework on eight tasks including classification, segmentation, retrieval, and semantic relatedness, achieving comparable or better performance with the zero-shot or few-shot settings. Our code is open on https://github.com/ChenXiaoFei-CS/KoBo.