mlsanityData to 14 Oct 2026

Using Multiple Instance Learning to Build Multimodal Representations

arXiv 2212.05561 ↗First posted 11 Dec 2022cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 11 Dec 2022.

Builds on

28 of 39 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2017Mask R-CNNcited by 7,277
  7. 2017Non-local Neural Networkscited by 2,293
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  10. 2018Attention-based Deep Multiple Instance Learningcited by 1,095
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
16 more
  1. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  2. 2020Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learningcited by 478
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  4. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  5. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  6. 2016Multiple Instance Learning: A Survey of Problem Characteristics and Applicationscited by 164
  7. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  8. 2014From Captions to Visual Concepts and Backcited by 109
  9. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  10. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  11. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  12. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  13. 2015Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learningcited by 38
  14. 2020PySBD: Pragmatic Sentence Boundary Disambiguationcited by 38
  15. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  16. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 2; 2026: 1.

7 papers.

  1. 2023Medical Vision Language Pretraining: A surveycited by 24
  2. 2024PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioningcited by 14
  3. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  4. 2023Sample-Specific Debiasing for Better Image-Text Modelscited by 1
  5. 2025Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Modelscited by 0
  6. 2025MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Predictioncited by 0
  7. 2026MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Imagescited by 0