mlsanityData to 7 Oct 2026

Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing

arXiv 2204.09817 ↗First posted 21 Apr 2022cs.CV cs.CLcited by 336

Not on the map. Why?

Cited by 115 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.

Key facts. Cited by 336 papers in the map (data through 7 Oct 2026). Builds on 59 of its 92 references that are in the map. First posted on arXiv 21 Apr 2022.

Builds on

59 of 92 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  11. 2018YOLOv3: An Incremental Improvementcited by 2,652
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  9. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  10. 2019Publicly Available Clinical BERT Embeddingscited by 995
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  13. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  14. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  15. 2017Large Batch Training of Convolutional Networkscited by 529
  16. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  17. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  18. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  19. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  20. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  21. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  22. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  23. 2018An efficient framework for learning sentence representationscited by 197
  24. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  25. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  26. 2019Clinically Accurate Chest X-Ray Report Generationcited by 138
  27. 2014From Captions to Visual Concepts and Backcited by 109
  28. 2020Adversarial Training for Large Neural Language Modelscited by 109
  29. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  30. 2016Learning Visual N-Grams from Web Datacited by 95
  31. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  32. 2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
  33. 2018Learning to Summarize Radiology Findingscited by 54
  34. 2019Align2Ground: Weakly Supervised Phrase Grounding Guided by Image-Caption Alignmentcited by 51
  35. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  36. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  37. 2021Relation-aware Instance Refinement for Weakly Supervised Visual Groundingcited by 35
  38. 2021REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rayscited by 34

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 60; 2024: 78; 2025: 92; 2026: 97.

336 papers.

Sort:

By area:

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  2. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  3. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  4. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  5. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  6. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  7. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  8. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  9. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  10. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  11. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 85
  12. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 81
38 more
  1. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  2. 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
  3. 2023Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-traincited by 75
  4. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 72
  5. 2024RaTEScore: A Metric for Radiology Report Generationcited by 65
  6. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  7. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  8. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  9. 2024Toward Visual Grounding: A Surveycited by 52
  10. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  11. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 44
  12. 2023M-FLAG: Medical Vision-Language Pre-training with Frozen Language Models and Latent Space Geometry Optimizationcited by 43
  13. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  14. 2023MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Imagescited by 41
  15. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  16. 2023CLIP in Medical Imaging: A Surveycited by 40
  17. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  18. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  19. 2024SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signalscited by 38
  20. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  21. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  22. 2023Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attributioncited by 34
  23. 2023Ariadne's Thread:Using Text Prompts to Improve Segmentation of Infected Areas from Chest X-ray imagescited by 33
  24. 2023Medical Phrase Grounding with Region-Phrase Context Contrastive Alignmentcited by 32
  25. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 32
  26. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  27. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  28. 2024BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Modelscited by 28
  29. 2023Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT modelscited by 27
  30. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  31. 2024Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Frameworkcited by 27
  32. 2023Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generationcited by 24
  33. 2023Medical Vision Language Pretraining: A surveycited by 24
  34. 2024Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Modelscited by 24
  35. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  36. 2023Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosiscited by 23
  37. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  38. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 23

50 most cited of 336 All 336 (JSON)