mlsanityData to 7 Oct 2026

PRIOR: Prototype Representation Joint Learning from Medical Images and Reports

arXiv 2307.12577 ↗First posted 24 Jul 2023cs.CVcited by 55

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive learning based vision-language joint pre-training has emerged as a successful representation learning strategy. In this paper, we present a prototype representation learning framework incorporating both global and local alignment between medical images and reports. In contrast to standard global multi-modality alignment methods, we employ a local alignment module for fine-grained representation. Furthermore, a cross-modality conditional reconstruction module is designed to interchange information across modalities in the training phase by reconstructing masked images and reports. For reconstructing long reports, a sentence-wise prototype memory bank is constructed, enabling the network to focus on low-level localized visual and high-level clinical linguistic features. Additionally, a non-auto-regressive generation paradigm is proposed for reconstructing non-sequential reports. Experimental results on five downstream tasks, including supervised classification, zero-shot classification, image-to-text retrieval, semantic segmentation, and object detection, show the proposed method outperforms other state-of-the-art methods across multiple datasets and under different dataset size settings. The code is available at https://github.com/QtacierP/PRIOR.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

49 of 65 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2020End-to-End Object Detection with Transformerscited by 7,384
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
37 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  10. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  13. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  14. 2021Grounded Language-Image Pre-trainingcited by 1,147
  15. 2019Publicly Available Clinical BERT Embeddingscited by 995
  16. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  17. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  18. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  20. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
  21. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  22. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  23. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  24. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  25. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  26. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  27. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  28. 2020Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancycited by 125
  29. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 101
  30. 2017Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hzcited by 91
  31. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  32. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  33. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  34. 2022End-to-End Transformer Based Model for Image Captioningcited by 37
  35. 2018Domain Adaptive Transfer Learning with Specialist Modelscited by 31
  36. 2021Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generationcited by 19
  37. 2022Aesthetic Text Logo Synthesis via Content-aware Layout Inferringcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 25; 2025: 12; 2026: 10.

55 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2023CLIP in Medical Imaging: A Surveycited by 40
  4. 2023Medical Vision Language Pretraining: A surveycited by 24
  5. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  6. 2024MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learningcited by 23
  7. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  8. 2024Eye-gaze Guided Multi-modal Alignment for Medical Representation Learningcited by 16
  9. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  10. 2024Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generationcited by 14
  11. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  12. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
38 more
  1. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  2. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  3. 2023Prototype-based Cross-Modal Object Trackingcited by 7
  4. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  5. 2024Multi-Modal CLIP-Informed Protein Editingcited by 6
  6. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  7. 2025Alleviating Textual Reliance in Medical Language-guided Segmentation via Prototype-driven Semantic Approximationcited by 6
  8. 2023Unified Medical Image Pre-training in Language-Guided Common Semantic Spacecited by 5
  9. 2024FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneitycited by 5
  10. 2024Topicwise Separable Sentence Retrieval for Medical Report Generationcited by 5
  11. 2024EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledgecited by 4
  12. 2024ProCNS: Progressive Prototype Calibration and Noise Suppression for Weakly-Supervised Medical Image Segmentationcited by 3
  13. 2024Masks and Manuscripts: Advancing Medical Pre-training with End-to-End Masking and Narrative Structuringcited by 3
  14. 2025Multimodal Medical Image Binding via Shared Text Embeddingscited by 3
  15. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Raycited by 2
  16. 2024Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classificationcited by 2
  17. 2024HYDEN: Hyperbolic Density Representations for Medical Images and Reportscited by 2
  18. 2024ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialoguecited by 2
  19. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  20. 2026Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysiscited by 2
  21. 2026A World Model of Radiologist Reading for Medical Image Representation Learningcited by 2
  22. 2024Multi-modality Regional Alignment Network for Covid X-Ray Survival Prediction and Report Generationcited by 1
  23. 2024Advancing Medical Radiograph Representation Learning: A Hybrid Pre-training Paradigm with Multilevel Semantic Granularitycited by 1
  24. 2024Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignmentcited by 1
  25. 2025Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Explorationcited by 1
  26. 2025Distribution-Based Masked Medical Vision-Language Model Using Structured Reportscited by 1
  27. 2025FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attentioncited by 1
  28. 2026Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rayscited by 1
  29. 2026T-DuMpRa: Teacher-guided Dual-path Multi-prototype Retrieval Augmented framework for fine-grained medical image classificationcited by 1
  30. 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-trainingcited by 1
  31. 2023Improving Medical Visual Representations via Radiology Report Generationcited by 0
  32. 2023ECAMP: Entity-centered Context-aware Medical Vision Language Pre-trainingcited by 0
  33. 2024Region-specific Risk Quantification for Interpretable Prognosis of COVID-19cited by 0
  34. 2024Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysiscited by 0
  35. 2025Progressive Local Alignment for Medical Multimodal Pre-trainingcited by 0
  36. 2025Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reportscited by 0
  37. 2025AGA: An adaptive group alignment framework for structured medical cross-modal representation learningcited by 0
  38. 2025Medical Report Generation: A Hierarchical Task Structure-Based Cross-Modal Causal Intervention Frameworkcited by 0

50 most cited of 55 All 55 (JSON)