mlsanityData to 7 Oct 2026

Joint Learning of Localized Representations from Medical Images and Reports

arXiv 2112.02889 ↗First posted 6 Dec 2021cs.CV cs.CL cs.LGcited by 81

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive learning has proven effective for pre-training image models on unlabeled data with promising results for tasks such as medical image classification. Using paired text (like radiological reports) during pre-training improves the results even further. Still, most existing methods target image classification downstream tasks and may not be optimal for localized tasks like semantic segmentation or object detection. We therefore propose Localized representation learning from Vision and Text (LoVT), to our best knowledge, the first text-supervised pre-training method that targets localized medical imaging tasks. Our method combines instance-level image-report contrastive learning with local contrastive learning on image region and report sentence representations. We evaluate LoVT and commonly used pre-training methods on an evaluation framework of 18 localized tasks on chest X-rays from five public datasets. LoVT performs best on 10 of the 18 studied tasks making it the preferred method of choice for localized tasks.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 77 of its 102 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

77 of 102 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2018YOLOv3: An Incremental Improvementcited by 2,652
  10. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  13. 2016Density estimation using Real NVPcited by 2,029
  14. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  15. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  16. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  17. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  18. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  19. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  21. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  22. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  23. 2016Colorful Image Colorizationcited by 1,113
  24. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  25. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  26. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  27. 2019Publicly Available Clinical BERT Embeddingscited by 995
  28. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  29. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  30. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  31. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  32. 2016Pixel Recurrent Neural Networkscited by 814
  33. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  34. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  35. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  36. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  37. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  38. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 22; 2024: 22; 2025: 19; 2026: 13.

81 papers.

Sort:

By area:

  1. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  2. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  3. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  4. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  5. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  6. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  7. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  8. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  9. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  10. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  11. 2023CLIP in Medical Imaging: A Surveycited by 40
  12. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 35
38 more
  1. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  2. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  3. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  4. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  5. 2023Medical Vision Language Pretraining: A surveycited by 24
  6. 2023From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusioncited by 19
  7. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
  8. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 17
  9. 2023Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regimecited by 15
  10. 2024Knowledge-enhanced Visual-Language Pretraining for Computational Pathologycited by 14
  11. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  12. 2023LIMITR: Leveraging Local Information for Medical Image-Text Representationcited by 13
  13. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  14. 2025Towards Scalable Language-Image Pre-training for 3D Medical Imagingcited by 13
  15. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  16. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  17. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  18. 2024DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentationcited by 9
  19. 2022Using Multiple Instance Learning to Build Multimodal Representationscited by 7
  20. 2023Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrievalcited by 6
  21. 2024Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-trainingcited by 6
  22. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  23. 2022The Role of Local Alignment and Uniformity in Image-Text Contrastive Learning on Medical Imagescited by 5
  24. 2023Unified Medical Image Pre-training in Language-Guided Common Semantic Spacecited by 5
  25. 2025Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECGcited by 5
  26. 2023MEDIMP: 3D Medical Images with clinical Prompts from limited tabular data for renal transplantationcited by 4
  27. 2023Local Contrastive Learning for Medical Image Recognitioncited by 4
  28. 2024Enhancing medical vision-language contrastive learning via inter-matching relation modellingcited by 4
  29. 2023LT-ViT: A Vision Transformer for multi-label Chest X-ray classificationcited by 3
  30. 2024Masks and Manuscripts: Advancing Medical Pre-training with End-to-End Masking and Narrative Structuringcited by 3
  31. 2024Image-aware Evaluation of Generated Medical Reportscited by 3
  32. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Raycited by 2
  33. 2024HYDEN: Hyperbolic Density Representations for Medical Images and Reportscited by 2
  34. 2025The Missing Piece: A Case for Pre-Training in 3D Medical Object Detectioncited by 2
  35. 2025Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generationcited by 2
  36. 2026Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysiscited by 2
  37. 2023Sample-Specific Debiasing for Better Image-Text Modelscited by 1
  38. 2024Optimizing Universal Lesion Segmentation: State Space Model-Guided Hierarchical Networks with Feature Importance Adjustmentcited by 1

50 most cited of 81 All 81 (JSON)