# A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports

arXiv: https://arxiv.org/abs/2009.01523 · HTML: https://mlsanity.com/paper/2009.01523/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 30 of 35)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts](https://mlsanity.com/paper/2302.08958/) (2023)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis](https://mlsanity.com/paper/2108.04938/) (2021)
- [DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries](https://mlsanity.com/paper/2205.01290/) (2022)
- [Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment](https://mlsanity.com/paper/2109.01949/) (2021)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Visual Grounding of Whole Radiology Reports for 3D CT Images](https://mlsanity.com/paper/2312.04794/) (2023)
- [Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcare](https://mlsanity.com/paper/2405.06784/) (2024)
- [Multi-modal Crowd Counting via a Broker Modality](https://mlsanity.com/paper/2407.07518/) (2024)
- [Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentation](https://mlsanity.com/paper/2305.12231/) (2023)
- [Variational Knowledge Distillation for Disease Classification in Chest X-Rays](https://mlsanity.com/paper/2103.10825/) (2021)
- [Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers](https://mlsanity.com/paper/2202.06076/) (2022)
- [Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives](https://mlsanity.com/paper/2111.04898/) (2021)
- [Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification](https://mlsanity.com/paper/2203.10622/) (2022)
- [TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning](https://mlsanity.com/paper/2111.10756/) (2021)
- [Visualizing and Explaining Language Models](https://mlsanity.com/paper/2205.10238/) (2022)
- [LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models](https://mlsanity.com/paper/2409.19540/) (2024)
- [Multimodal Machine Learning in Precision Health](https://mlsanity.com/paper/2204.04777/) (2022)
- [Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning](https://mlsanity.com/paper/2404.06057/) (2024)
- [The Ability of Image-Language Explainable Models to Resemble Domain Expertise](https://mlsanity.com/paper/2209.09310/) (2022)
- [AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease](https://mlsanity.com/paper/2212.06042/) (2022)

## Abstract (arXiv)

Joint image-text embedding extracted from medical images and associated contextual reports is the bedrock for most biomedical vision-and-language (V+L) tasks, including medical visual question answering, clinical image-text retrieval, clinical report auto-generation. In this study, we adopt four pre-trained V+L models: LXMERT, VisualBERT, UNIER and PixelBERT to learn multimodal representation from MIMIC-CXR radiographs and associated reports. The extrinsic evaluation on OpenI dataset shows that in comparison to the pioneering CNN-RNN model, the joint embedding learned by pre-trained V+L models demonstrate performance improvement in the thoracic findings classification task. We conduct an ablation study to analyze the contribution of certain model components and validate the advantage of joint embedding over text-only embedding. We also visualize attention maps to illustrate the attention mechanism of V+L models.