# Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models

arXiv: https://arxiv.org/abs/2306.08000 · HTML: https://mlsanity.com/paper/2306.08000/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 8 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)

## Built on by (most cited first, 6 of 6)

- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports](https://mlsanity.com/paper/2407.05758/) (2024)
- [Vision-language models for decoding provider attention during neonatal resuscitation](https://mlsanity.com/paper/2404.01207/) (2024)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)
- [Improving Medical Visual Representations via Radiology Report Generation](https://mlsanity.com/paper/2310.19635/) (2023)

## Abstract (arXiv)

Recent advances in zero-shot learning have enabled the use of paired image-text data to replace structured labels, replacing the need for expert annotated datasets. Models such as CLIP-based CheXzero utilize these advancements in the domain of chest X-ray interpretation. We hypothesize that domain pre-trained models such as CXR-BERT, BlueBERT, and ClinicalBERT offer the potential to improve the performance of CLIP-like models with specific domain knowledge by replacing BERT weights at the cost of breaking the original model's alignment. We evaluate the performance of zero-shot classification models with domain-specific pre-training for detecting low-prevalence pathologies. Even though replacing the weights of the original CLIP-BERT degrades model performance on commonly found pathologies, we show that pre-trained text towers perform exceptionally better on low-prevalence diseases. This motivates future ensemble models with a combination of differently trained language models for maximal performance.