mlsanityData to 7 Oct 2026

Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles

arXiv 2103.05121 ↗First posted 8 Mar 2021cs.CV cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

We present ARCH, a computational pathology (CP) multiple instance captioning dataset to facilitate dense supervision of CP tasks. Existing CP datasets focus on narrow tasks; ARCH on the other hand contains dense diagnostic and morphological descriptions for a range of stains, tissue types and pathologies. Using intrinsic dimensionality estimation, we show that ARCH is the only CP dataset to (ARCH-)rival its computer vision analog MS-COCO Captions. We conjecture that an encoder pre-trained on dense image captions learns transferable representations for most CP tasks. We support the conjecture with evidence that ARCH representation transfers to a variety of pathology sub-tasks better than ImageNet features or representations obtained via self-supervised or multi-task learning on pathology images alone. We release our best model and invite other researchers to test it on their CP tasks.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

33 of 64 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2018Attention-based Deep Multiple Instance Learningcited by 1,077
  9. 2018Panoptic Segmentationcited by 796
  10. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 738
  11. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  12. 2018Do Better ImageNet Models Transfer Better?cited by 560
21 more
  1. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  2. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  3. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  4. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  5. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  6. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  7. 2017A Review on Deep Learning Techniques Applied to Semantic Segmentationcited by 122
  8. 2017Self-supervised Learning of Motion Capturecited by 97
  9. 2018MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Imagescited by 87
  10. 2018The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approachescited by 74
  11. 2019Automated Gleason Grading of Prostate Biopsies using Deep Learningcited by 69
  12. 2018Neural Image Compression for Gigapixel Histopathology Image Analysiscited by 65
  13. 2018Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challengecited by 61
  14. 2017MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Networkcited by 56
  15. 2018Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learningcited by 52
  16. 2019Similar Image Search for Histopathology: SMILYcited by 41
  17. 2019Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Codingcited by 35
  18. 2018Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancercited by 34
  19. 2020Multi-task pre-training of deep neural networks for digital pathologycited by 14
  20. 2020Extending Unsupervised Neural Image Compression With Supervised Multitask Learningcited by 12
  21. 2019Probing the State of the Art: A Critical Look at Visual Representation Evaluationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 15; 2024: 21; 2025: 10; 2026: 7.

58 papers.

Sort:

By area:

  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  4. 2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
  5. 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 104
  6. 2024PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathologycited by 80
  7. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  8. 2024HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interactioncited by 43
  9. 2022One Model is All You Need: Multi-Task Learning Enables Simultaneous Histology Image Segmentation and Classificationcited by 38
  10. 2024WSI-VQA: Interpreting Whole Slide Images by Generative Visual Question Answeringcited by 34
  11. 2023WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Imagescited by 31
  12. 2024PathAlign: A vision-language model for whole slide images in histopathologycited by 29
38 more
  1. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  2. 2024Multistain Pretraining for Slide Representation Learning in Pathologycited by 27
  3. 2022Inference of captions from histopathological patchescited by 26
  4. 2021Self-Supervised Representation Learning using Visual Field Expansion on Digital Pathologycited by 25
  5. 2023Medical Vision Language Pretraining: A surveycited by 24
  6. 2024STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomicscited by 22
  7. 2024Knowledge-enhanced Pretraining for Vision-language Pathology Foundation Model on Cancer Diagnosiscited by 21
  8. 2024CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignmentcited by 20
  9. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  10. 2023A Foundational Multimodal Vision Language AI Assistant for Human Pathologycited by 19
  11. 2023AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Imagescited by 18
  12. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14
  13. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  14. 2024PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioningcited by 13
  15. 2023Divert More Attention to Vision-Language Object Trackingcited by 11
  16. 2024Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysiscited by 10
  17. 2023Automatic Report Generation for Histopathology images using pre-trained Vision Transformerscited by 9
  18. 2024A New Era in Computational Pathology: A Survey on Foundation and Vision-Language Modelscited by 9
  19. 2025Multi-Modal Foundation Models for Computational Pathology: A Surveycited by 9
  20. 2023What a Whole Slide Image Can Tell? Subtype-guided Masked Transformer for Pathological Image Captioningcited by 8
  21. 2022LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archivescited by 5
  22. 2025Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challengescited by 5
  23. 2024Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learningcited by 4
  24. 2025MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classificationcited by 3
  25. 2025MedicalNarratives: Connecting Medical Vision and Language with Localized Narrativescited by 2
  26. 2025Histopathology Multi-modal Embedding for Pathology Composed Retrievalcited by 2
  27. 2023Masked conditional variational autoencoders for chromosome straighteningcited by 1
  28. 2024Multi-Scale Heterogeneity-Aware Hypergraph Representation for Histopathology Whole Slide Imagescited by 1
  29. 2024Leveraging image captions for selective whole slide image annotationcited by 1
  30. 2024Advancing Histopathology with Deep Learning Under Data Scarcity: A Decade in Reviewcited by 1
  31. 2024Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answeringcited by 1
  32. 2025HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Taskscited by 1
  33. 2025A Unified Low-level Foundation Model for Enhancing Pathology Image Qualitycited by 1
  34. 2025PathoHR: Hierarchical Reasoning for Vision-Language Models in Pathologycited by 1
  35. 2026A Deployment-Friendly Foundational Framework for Efficient Computational Pathologycited by 1
  36. 2026Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generationcited by 1
  37. 2026Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarkscited by 1
  38. 2023Automatic Report Generation for Histopathology images using pre-trained Vision Transformers and BERTcited by 0

50 most cited of 58 All 58 (JSON)