# Endo-CLIP: Progressive Self-Supervised Pre-training on Raw Colonoscopy Records

arXiv: https://arxiv.org/abs/2505.09435 · HTML: https://mlsanity.com/paper/2505.09435/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 14 May 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [A Self-Supervised Descriptor for Image Copy Detection](https://mlsanity.com/paper/2202.10261/) (2022)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2306.16741/) (2023)
- [Swin MAE: Masked Autoencoders for Small Datasets](https://mlsanity.com/paper/2212.13805/) (2022)
- [Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training](https://mlsanity.com/paper/2307.07246/) (2023)
- [Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models](https://mlsanity.com/paper/2310.11173/) (2023)

## Built on by (most cited first, 2 of 2)

- [Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos](https://mlsanity.com/paper/2603.25645/) (2026)
- [HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space](https://mlsanity.com/paper/2606.31245/) (2026)

## Abstract (arXiv)

Pre-training on image-text colonoscopy records offers substantial potential for improving endoscopic image analysis, but faces challenges including non-informative background images, complex medical terminology, and ambiguous multi-lesion descriptions. We introduce Endo-CLIP, a novel self-supervised framework that enhances Contrastive Language-Image Pre-training (CLIP) for this domain. Endo-CLIP's three-stage framework--cleansing, attunement, and unification--addresses these challenges by (1) removing background frames, (2) leveraging large language models to extract clinical attributes for fine-grained contrastive learning, and (3) employing patient-level cross-attention to resolve multi-polyp ambiguities. Extensive experiments demonstrate that Endo-CLIP significantly outperforms state-of-the-art pre-training methods in zero-shot and few-shot polyp detection and classification, paving the way for more accurate and clinically relevant endoscopic analysis.