# Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

arXiv: https://arxiv.org/abs/2202.06076 · HTML: https://mlsanity.com/paper/2202.06076/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 12 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/1803.02315/) (2018)
- [Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment](https://mlsanity.com/paper/2008.09884/) (2020)
- [A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports](https://mlsanity.com/paper/2009.01523/) (2020)
- [On Adversarial Examples for Biomedical NLP Tasks](https://mlsanity.com/paper/2004.11157/) (2020)
- [INSIDE: Steering Spatial Attention with Non-Imaging Information in CNNs](https://mlsanity.com/paper/2008.10418/) (2020)
- [Cross-Modal Information Maximization for Medical Imaging: CMIM](https://mlsanity.com/paper/2010.10593/) (2020)
- [Conditioning Convolutional Segmentation Architectures with Non-Imaging Data](https://mlsanity.com/paper/1907.12330/) (2019)

## Built on by (most cited first, 8 of 8)

- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Privacy Distillation: Reducing Re-identification Risk of Multimodal Diffusion Models](https://mlsanity.com/paper/2306.01322/) (2023)
- [Dynamic Multimodal Information Bottleneck for Multimodality Classification](https://mlsanity.com/paper/2311.01066/) (2023)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data](https://mlsanity.com/paper/2309.15529/) (2023)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [Towards Precision Healthcare: Robust Fusion of Time Series and Image Data](https://mlsanity.com/paper/2405.15442/) (2024)

## Abstract (arXiv)

When a clinician refers a patient for an imaging exam, they include the reason (e.g. relevant patient history, suspected disease) in the scan request; this appears as the indication field in the radiology report. The interpretation and reporting of the image are substantially influenced by this request text, steering the radiologist to focus on particular aspects of the image. We use the indication field to drive better image classification, by taking a transformer network which is unimodally pre-trained on text (BERT) and fine-tuning it for multimodal classification of a dual image-text input. We evaluate the method on the MIMIC-CXR dataset, and present ablation studies to investigate the effect of the indication field on the classification performance. The experimental results show our approach achieves 87.8 average micro AUROC, outperforming the state-of-the-art methods for unimodal (84.4) and multimodal (86.0) classification. Our code is available at https://github.com/jacenkow/mmbt.