# CLIP-IT: CLIP-based Pairing for Histology Images Classification

arXiv: https://arxiv.org/abs/2504.16181 · HTML: https://mlsanity.com/paper/2504.16181/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction](https://mlsanity.com/paper/2403.05396/) (2024)
- [WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images](https://mlsanity.com/paper/2311.16480/) (2023)
- [MoME: Mixture of Multimodal Experts for Cancer Survival Prediction](https://mlsanity.com/paper/2406.09696/) (2024)
- [PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning](https://mlsanity.com/paper/2403.08967/) (2024)
- [Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology](https://mlsanity.com/paper/2404.19113/) (2024)
- [PathoTune: Adapting Visual Foundation Model to Pathological Specialists](https://mlsanity.com/paper/2403.16497/) (2024)
- [Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models](https://mlsanity.com/paper/2407.14426/) (2024)
- [Disease-informed Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.15728/) (2024)
- [Towards a text-based quantitative and explainable histopathology image analysis](https://mlsanity.com/paper/2407.07360/) (2024)
- [Spatial Transcriptomics Analysis of Zero-shot Gene Expression Prediction](https://mlsanity.com/paper/2401.14772/) (2024)
- [Prompting Whole Slide Image Based Genetic Biomarker Prediction](https://mlsanity.com/paper/2407.09540/) (2024)
- [Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations](https://mlsanity.com/paper/2407.17726/) (2024)
- [Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning](https://mlsanity.com/paper/2403.14616/) (2024)
- [Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning](https://mlsanity.com/paper/2406.13979/) (2024)
- [PixelCAM: Pixel Class Activation Mapping for Histology Image Classification and ROI Localization](https://mlsanity.com/paper/2503.24135/) (2025)
- [DSCENet: Dynamic Screening and Clinical-Enhanced Multimodal Fusion for MPNs Subtype Classification](https://mlsanity.com/paper/2407.08167/) (2024)

## Built on by (most cited first, 1 of 1)

- [Selective Prediction Forgetting for Adaptation of Weakly Supervised Localization of Histology Images](https://mlsanity.com/paper/2603.12468/) (2026)

## Abstract (arXiv)

Multimodal learning has shown promise in medical imaging, combining complementary modalities like images and text. Vision-language models (VLMs) capture rich diagnostic cues but often require large paired datasets and prompt- or text-based inference, limiting their practicality due to annotation cost, privacy, and compute demands. Crucially, available free unpaired external text, like pathology reports, can still provide complementary diagnostic cues if semantically relevant content is retrievable per image. To address this, we introduce CLIP-IT, a novel framework that relies on rich unpaired text reports. Specifically, CLIP-IT uses a CLIP model pre-trained on histology image-text pairs from a separate dataset to retrieve the most relevant unpaired textual report for each image in the downstream unimodal dataset. These reports, sourced from the same disease domain and tissue type, form pseudo-pairs that reflect shared clinical semantics rather than exact alignment. Knowledge from these texts is distilled into the vision model during training, while LoRA-based adaptation mitigates the semantic gap between unaligned modalities. At inference, only the vision model is used, keeping overhead low while still benefiting from multimodal training without requiring paired data in the downstream dataset. Experiments on histology image datasets confirm that CLIP-IT consistently improves classification accuracy over both unimodal and multimodal CLIP-based baselines in most cases, without the burden of per-dataset paired annotation or inference-time complexity.