mlsanityData to 7 Oct 2026

A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports

arXiv 2607.28466 ↗First posted 30 Jul 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Vision-language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision-language foundation model trained on 125,756 lesion-level image-text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image-text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings. On benign-versus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 46 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

17 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026GI-Bench: A Panoramic Benchmark Revealing the Knowledge-Experience Dissociation of Multimodal Large Language Models in Gastrointestinal Endoscopy Against Clinical Standards13 Jan

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  9. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
  10. 2016A Benchmark for Endoluminal Scene Segmentation of Colonoscopy Imagescited by 220
  11. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  12. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
4 more
  1. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  2. 2021A multi-centre polyp detection and segmentation dataset for generalisability assessmentcited by 63
  3. 2025Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Modelscited by 3
  4. 2024EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosiscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
  2. 2026EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment
  3. 2026Colon-Bench: An Agentic Workflow for Scalable Dense Lesion Annotation in Full-Procedure Colonoscopy Videos
  4. 2025LDP: Parameter-Efficient Fine-Tuning of Multimodal LLM for Medical Report Generation
  5. 2026RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing
  6. 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
  7. 2026LUX: A Lesion-Aware Graph-Conditioned Visual - Language Architecture for Explainable Endoscopic Captioning
  8. 2025Colon-X: Advancing Intelligent Colonoscopy toward Clinical Reasoning