# Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs

arXiv: https://arxiv.org/abs/2107.06563 · HTML: https://mlsanity.com/paper/2107.06563/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [Multi-Label Zero-Shot Learning with Structured Knowledge Graphs](https://mlsanity.com/paper/1711.06526/) (2017)
- [Improving zero-shot learning by mitigating the hubness problem](https://mlsanity.com/paper/1412.6568/) (2014)
- [Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision](https://mlsanity.com/paper/1709.01148/) (2017)
- [Transductive Zero-Shot Learning for 3D Point Cloud Classification](https://mlsanity.com/paper/1912.07161/) (2019)
- [Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification](https://mlsanity.com/paper/1801.09927/) (2018)
- [Preserving Semantic Relations for Zero-Shot Learning](https://mlsanity.com/paper/1803.03049/) (2018)
- [Learning to diagnose from scratch by exploiting dependencies among labels](https://mlsanity.com/paper/1710.10501/) (2017)
- [Synthesizing the Unseen for Zero-shot Object Detection](https://mlsanity.com/paper/2010.09425/) (2020)
- [From Zero-shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis](https://mlsanity.com/paper/1705.01782/) (2017)
- [Zero-Shot Learning and its Applications from Autonomous Vehicles to COVID-19 Diagnosis: A Review](https://mlsanity.com/paper/2004.14143/) (2020)

## Built on by (most cited first, 5 of 5)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [Looking Beyond What You See: An Empirical Analysis on Subgroup Intersectional Fairness for Multi-label Chest X-ray Classification Using Social Determinants of Racial Health Inequities](https://mlsanity.com/paper/2403.18196/) (2024)
- [XDT-CXR: Investigating Cross-Disease Transferability in Zero-Shot Binary Classification of Chest X-Rays](https://mlsanity.com/paper/2408.11493/) (2024)

## Abstract (arXiv)

Despite the success of deep neural networks in chest X-ray (CXR) diagnosis, supervised learning only allows the prediction of disease classes that were seen during training. At inference, these networks cannot predict an unseen disease class. Incorporating a new class requires the collection of labeled data, which is not a trivial task, especially for less frequently-occurring diseases. As a result, it becomes inconceivable to build a model that can diagnose all possible disease classes. Here, we propose a multi-label generalized zero shot learning (CXR-ML-GZSL) network that can simultaneously predict multiple seen and unseen diseases in CXR images. Given an input image, CXR-ML-GZSL learns a visual representation guided by the input's corresponding semantics extracted from a rich medical text corpus. Towards this ambitious goal, we propose to map both visual and semantic modalities to a latent feature space using a novel learning objective. The objective ensures that (i) the most relevant labels for the query image are ranked higher than irrelevant labels, (ii) the network learns a visual representation that is aligned with its semantics in the latent feature space, and (iii) the mapped semantics preserve their original inter-class representation. The network is end-to-end trainable and requires no independent pre-training for the offline feature extractor. Experiments on the NIH Chest X-ray dataset show that our network outperforms two strong baselines in terms of recall, precision, f1 score, and area under the receiver operating characteristic curve. Our code is publicly available at: https://github.com/nyuad-cai/CXR-ML-GZSL.git