mlsanityData to 7 Oct 2026

Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment

arXiv 2109.01949 ↗First posted 4 Sep 2021cs.LG cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 4 Sep 2021.

Builds on

22 of 27 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
10 more
  1. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  2. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  3. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  4. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  5. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  6. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  7. 2014From Captions to Visual Concepts and Backcited by 109
  8. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  9. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  10. 2020Attention based Writer Independent Handwriting Verificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 4; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generationcited by 25
  2. 2025O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Modelscited by 14
  3. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  4. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  5. 2024Looking Beyond What You See: An Empirical Analysis on Subgroup Intersectional Fairness for Multi-label Chest X-ray Classification Using Social Determinants of Racial Health Inequitiescited by 3
  6. 2025A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Modelscited by 3
  7. 2023A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Imagescited by 2
  8. 2024Tumor Location-weighted MRI-Report Contrastive Learning: A Framework for Improving the Explainability of Pediatric Brain Tumor Diagnosiscited by 1
  9. 2025Bridging Electronic Health Records and Clinical Texts: Contrastive Learning for Enhanced Clinical Taskscited by 1
  10. 2022A Bayesian Detect to Track System for Robust Visual Object Tracking and Semi-Supervised Model Learningcited by 0
  11. 2022That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Datacited by 0
  12. 2025Anatomy-Aware Conditional Image-Text Retrievalcited by 0
1 more
  1. 2026Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imagingcited by 0