mlsanityData to 14 Oct 2026

Quilt-1M: One Million Image-Text Pairs for Histopathology

arXiv 2306.11207 ↗First posted 20 Jun 2023cs.CV cs.CL cs.LGcited by 194

Not on the map. Why?

Cited by 80 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering 1,087 hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of 802, 144 image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around 200K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with 1M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across 13 diverse patch-level datasets of 8 different sub-pathologies and cross-modal retrieval tasks.

Key facts. Cited by 194 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

42 of 73 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  10. 2022A ConvNet for the 2020scited by 3,579
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  12. 2019Towards VQA Models That Can Readcited by 1,718
30 more
  1. 2018Datasheets for Datasetscited by 1,386
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2016A Diagram Is Worth A Dozen Imagescited by 804
  4. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  5. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  6. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  7. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  8. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  9. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  10. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  11. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  12. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  13. 2019Connecting Vision and Language with Localized Narrativescited by 226
  14. 2022Is GPT-3 a Good Data Annotator?cited by 208
  15. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  16. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  17. 2018BACH: Grand Challenge on Breast Cancer Histology Imagescited by 131
  18. 2019Lung and Colon Cancer Histopathological Image Dataset (LC25000)cited by 89
  19. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  20. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  21. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 68
  22. 2021A Petri Dish for Histopathology Image Analysiscited by 64
  23. 2022BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pixcited by 61
  24. 2021Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articlescited by 58
  25. 2023Connecting Vision and Language with Video Localized Narrativescited by 24
  26. 2022PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domaincited by 12
  27. 2022ACROBAT -- a multi-stain breast cancer histological whole-slide-image data set from routine diagnostics for computational pathologycited by 10
  28. 2016Large-Scale Query-by-Image Video Retrieval Using Bloom Filterscited by 6
  29. 2020SlideImages: A Dataset for Educational Image Classificationcited by 5
  30. 2022Multi-modal Masked Autoencoders Learn Compositional Histopathological Representationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 58; 2025: 78; 2026: 50.

194 papers.

Sort:

By area:

  1. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 230
  2. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 161
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  5. 2023Virchow: A Million-Slide Digital Pathology Foundation Modelcited by 90
  6. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  7. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  8. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  9. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 81
  10. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  11. 2025Vision-Language Models Do Not Understand Negationcited by 72
  12. 2024PathMMU: A Massive Multimodal Expert-Level Benchmark for Understanding and Reasoning in Pathologycited by 43
38 more
  1. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 42
  2. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  3. 2023Learned representation-guided diffusion models for large-image generationcited by 40
  4. 2023CLIP in Medical Imaging: A Surveycited by 40
  5. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  6. 2025PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathologycited by 39
  7. 2024CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathologycited by 34
  8. 2024BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrievalcited by 30
  9. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 30
  10. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  11. 2025CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logiccited by 29
  12. 2024PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaborationcited by 28
  13. 2025ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classificationcited by 28
  14. 2025Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatologycited by 28
  15. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  16. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 27
  17. 2023Medical Vision Language Pretraining: A surveycited by 24
  18. 2024BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversitycited by 24
  19. 2024Few-shot Adaptation of Medical Vision-Language Modelscited by 24
  20. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  21. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 24
  22. 2024STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomicscited by 23
  23. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  24. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 21
  25. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosiscited by 21
  26. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  27. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 18
  28. 2024PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learningcited by 17
  29. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  30. 2024BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learningcited by 16
  31. 2024PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understandingcited by 16
  32. 2025A Survey on Computational Pathology Foundation Models: Datasets, Adaptation Strategies, and Evaluation Taskscited by 16
  33. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
  34. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  35. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understandingcited by 15
  36. 2026MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Imagescited by 15
  37. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14
  38. 2024Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathologycited by 14

50 most cited of 194 All 194 (JSON)