mlsanityData to 7 Oct 2026

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

arXiv 2202.06076 ↗First posted 12 Feb 2022cs.CV cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

When a clinician refers a patient for an imaging exam, they include the reason (e.g. relevant patient history, suspected disease) in the scan request; this appears as the indication field in the radiology report. The interpretation and reporting of the image are substantially influenced by this request text, steering the radiologist to focus on particular aspects of the image. We use the indication field to drive better image classification, by taking a transformer network which is unimodally pre-trained on text (BERT) and fine-tuning it for multimodal classification of a dual image-text input. We evaluate the method on the MIMIC-CXR dataset, and present ablation studies to investigate the effect of the indication field on the classification performance. The experimental results show our approach achieves 87.8 average micro AUROC, outperforming the state-of-the-art methods for unimodal (84.4) and multimodal (86.0) classification. Our code is available at https://github.com/jacenkow/mmbt.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 13 of its 24 references that are in the map. First posted on arXiv 12 Feb 2022.

Builds on

13 of 24 references.

References · 13

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  6. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  7. 2018Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classificationcited by 57
  8. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  9. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  10. 2020On Adversarial Examples for Biomedical NLP Taskscited by 8
  11. 2020INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNscited by 6
  12. 2020Cross-Modal Information Maximization for Medical Imaging: CMIMcited by 3
1 more
  1. 2019Conditioning Convolutional Segmentation Architectures with Non-Imaging Datacited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 1.

8 papers.

  1. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  2. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  3. 2023Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Modelscited by 18
  4. 2023Dynamic Multimodal Information Bottleneck for Multimodality Classificationcited by 14
  5. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  6. 2023Missing-modality Enabled Multi-modal Fusion Architecture for Medical Datacited by 8
  7. 2023TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Predictioncited by 3
  8. 2024Towards Precision Healthcare: Robust Fusion of Time Series and Image Datacited by 0