# HydraViT: Adaptive Multi-Branch Transformer for Multi-Label Disease Classification from Chest X-ray Images

arXiv: https://arxiv.org/abs/2310.06143 · HTML: https://mlsanity.com/paper/2310.06143/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 69 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Adaptive Diffusion Priors for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2207.05876/) (2022)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/1803.02315/) (2018)
- [Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels](https://mlsanity.com/paper/2005.12734/) (2020)
- [Generation of Anonymous Chest Radiographs Using Latent Diffusion Models for Training Thoracic Abnormality Classification Systems](https://mlsanity.com/paper/2211.01323/) (2022)
- [Learning to diagnose from scratch by exploiting dependencies among labels](https://mlsanity.com/paper/1710.10501/) (2017)
- [Chest X-rays Classification: A Multi-Label and Fine-Grained Problem](https://mlsanity.com/paper/1807.07247/) (2018)
- [BolT: Fused Window Transformers for fMRI Time Series Analysis](https://mlsanity.com/paper/2205.11578/) (2022)
- [Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays](https://mlsanity.com/paper/1807.06067/) (2018)
- [Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks](https://mlsanity.com/paper/1804.07839/) (2018)
- [Region Proposals for Saliency Map Refinement for Weakly-supervised Disease Localisation and Classification](https://mlsanity.com/paper/2005.10550/) (2020)
- [Deep Hiearchical Multi-Label Classification Applied to Chest X-Ray Abnormality Taxonomies](https://mlsanity.com/paper/2009.05609/) (2020)
- [Content-Based Medical Image Retrieval with Opponent Class Adaptive Margin Loss](https://mlsanity.com/paper/2211.15371/) (2022)
- [Deep Clustering via Center-Oriented Margin Free-Triplet Loss for Skin Lesion Detection in Highly Imbalanced Datasets](https://mlsanity.com/paper/2204.02275/) (2022)
- [Weakly Supervised Thoracic Disease Localization via Disease Masks](https://mlsanity.com/paper/2101.09915/) (2021)
- [A Weakly Supervised Adaptive DenseNet for Classifying Thoracic Diseases and Identifying Abnormalities](https://mlsanity.com/paper/1807.01257/) (2018)

## Built on by (most cited first, 6 of 6)

- [DenoMamba: A fused state-space model for low-dose CT denoising](https://mlsanity.com/paper/2409.13094/) (2024)
- [Self-Supervised Multi-Modality Learning for Multi-Label Skin Lesion Classification](https://mlsanity.com/paper/2310.18583/) (2023)
- [CLARiTy: A Vision Transformer for Multi-Label Classification and Weakly-Supervised Localization of Chest X-ray Pathologies](https://mlsanity.com/paper/2512.16700/) (2025)
- [Estimating Conditional Covariance between labels for Multilabel Data](https://mlsanity.com/paper/2508.18951/) (2025)
- [PulmoSight-XAI: An Explainable Multi-View Attention Ensemble with Gradient Boosting Meta-Learning for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/2607.04478/) (2026)
- [Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/2609.09185/) (2026)

## Abstract (arXiv)

Chest X-ray is an essential diagnostic tool in the identification of chest diseases given its high sensitivity to pathological abnormalities in the lungs. However, image-driven diagnosis is still challenging due to heterogeneity in size and location of pathology, as well as visual similarities and co-occurrence of separate pathology. Since disease-related regions often occupy a relatively small portion of diagnostic images, classification models based on traditional convolutional neural networks (CNNs) are adversely affected given their locality bias. While CNNs were previously augmented with attention maps or spatial masks to guide focus on potentially critical regions, learning localization guidance under heterogeneity in the spatial distribution of pathology is challenging. To improve multi-label classification performance, here we propose a novel method, HydraViT, that synergistically combines a transformer backbone with a multi-branch output module with learned weighting. The transformer backbone enhances sensitivity to long-range context in X-ray images, while using the self-attention mechanism to adaptively focus on task-critical regions. The multi-branch output module dedicates an independent branch to each disease label to attain robust learning across separate disease classes, along with an aggregated branch across labels to maintain sensitivity to co-occurrence relationships among pathology. Experiments demonstrate that, on average, HydraViT outperforms competing attention-guided methods by 1.2%, region-guided methods by 1.4%, and semantic-guided methods by 1.0% in multi-label classification performance.