# CLIP-Lung: Textual Knowledge-Guided Lung Nodule Malignancy Prediction

arXiv: https://arxiv.org/abs/2304.08013 · HTML: https://mlsanity.com/paper/2304.08013/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 11 of 11)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Unimodal probability distributions for deep ordinal classification](https://mlsanity.com/paper/1705.05278/) (2017)
- [Learning from Ambiguous Labels for Lung Nodule Malignancy Prediction](https://mlsanity.com/paper/2104.11436/) (2021)
- [Shape and Margin-Aware Lung Nodule Classification in Low-dose CT Images via Soft Activation Mapping](https://mlsanity.com/paper/1810.12494/) (2018)
- [Meta Ordinal Regression Forest for Medical Image Classification with Ordinal Labels](https://mlsanity.com/paper/2203.07725/) (2022)

## Built on by (most cited first, 16 of 16)

- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [FairCLIP: Harnessing Fairness in Vision-Language Learning](https://mlsanity.com/paper/2403.19949/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Robust COVID-19 Detection in CT Images with CLIP](https://mlsanity.com/paper/2403.08947/) (2024)
- [TIME: TabPFN-Integrated Multimodal Engine for Robust Tabular-Image Learning](https://mlsanity.com/paper/2506.00813/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2403.09948/) (2024)
- [Foundation Models in Medical Imaging: A Review and Outlook](https://mlsanity.com/paper/2506.09095/) (2025)
- [AutoRad-Lung: A Radiomic-Guided Prompting Autoregressive Vision-Language Model for Lung Nodule Malignancy Prediction](https://mlsanity.com/paper/2503.20662/) (2025)
- [Vision-Language Model-Based Semantic-Guided Imaging Biomarker for Lung Nodule Malignancy Prediction](https://mlsanity.com/paper/2504.21344/) (2025)
- [Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability](https://mlsanity.com/paper/2411.04008/) (2024)
- [Prompt Mechanisms in Medical Imaging: A Comprehensive Survey](https://mlsanity.com/paper/2507.01055/) (2025)
- [DesignCLIP: Multimodal Learning with CLIP for Design Patent Understanding](https://mlsanity.com/paper/2508.15297/) (2025)
- [LEAML: Label-Efficient Adaptation to Out-of-Distribution Visual Tasks for Multimodal Large Language Models](https://mlsanity.com/paper/2510.03232/) (2025)
- [MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration](https://mlsanity.com/paper/2604.11197/) (2026)
- [Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning](https://mlsanity.com/paper/2606.11682/) (2026)

## Abstract (arXiv)

Lung nodule malignancy prediction has been enhanced by advanced deep-learning techniques and effective tricks. Nevertheless, current methods are mainly trained with cross-entropy loss using one-hot categorical labels, which results in difficulty in distinguishing those nodules with closer progression labels. Interestingly, we observe that clinical text information annotated by radiologists provides us with discriminative knowledge to identify challenging samples. Drawing on the capability of the contrastive language-image pre-training (CLIP) model to learn generalized visual representations from text annotations, in this paper, we propose CLIP-Lung, a textual knowledge-guided framework for lung nodule malignancy prediction. First, CLIP-Lung introduces both class and attribute annotations into the training of the lung nodule classifier without any additional overheads in inference. Second, we designed a channel-wise conditional prompt (CCP) module to establish consistent relationships between learnable context prompts and specific feature maps. Third, we align image features with both class and attribute features via contrastive learning, rectifying false positives and false negatives in latent space. The experimental results on the benchmark LIDC-IDRI dataset have demonstrated the superiority of CLIP-Lung, both in classification performance and interpretability of attention maps.