# Learning Hierarchical Attention for Weakly-supervised Chest X-Ray Abnormality Localization and Diagnosis

arXiv: https://arxiv.org/abs/2112.12349 · HTML: https://mlsanity.com/paper/2112.12349/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 23 Dec 2021.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/1803.02315/) (2018)
- [Interpreting Chest X-rays via CNNs that Exploit Hierarchical Disease Dependencies and Uncertainty Labels](https://mlsanity.com/paper/2005.12734/) (2020)
- [Thoracic Disease Identification and Localization with Limited Supervision](https://mlsanity.com/paper/1711.06373/) (2017)
- [Diagnose like a Radiologist: Attention Guided Convolutional Neural Network for Thorax Disease Classification](https://mlsanity.com/paper/1801.09927/) (2018)
- [Learning to recognize Abnormalities in Chest X-Rays with Location-Aware Dense Networks](https://mlsanity.com/paper/1803.04565/) (2018)
- [Weakly Supervised Medical Diagnosis and Localization from Multiple Resolutions](https://mlsanity.com/paper/1803.07703/) (2018)
- [Multi-label Thoracic Disease Image Classification with Cross-Attention Networks](https://mlsanity.com/paper/2007.10859/) (2020)
- [Weakly Supervised Deep Learning for Thoracic Disease Classification and Localization on Chest X-rays](https://mlsanity.com/paper/1807.06067/) (2018)
- [Sharpen Focus: Learning with Attention Separability and Consistency](https://mlsanity.com/paper/1811.07484/) (2018)
- [ProNet: Learning to Propose Object-specific Boxes for Cascaded Neural Networks](https://mlsanity.com/paper/1511.03776/) (2015)
- [Weakly Supervised Universal Fracture Detection in Pelvic X-rays](https://mlsanity.com/paper/1909.02077/) (2019)

## Built on by (most cited first, 24 of 24)

- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [Improving deep neural network generalization and robustness to background bias via layer-wise relevance propagation optimization](https://mlsanity.com/paper/2202.00232/) (2022)
- [MProtoNet: A Case-Based Interpretable Model for Brain Tumor Classification with 3D Multi-parametric Magnetic Resonance Imaging](https://mlsanity.com/paper/2304.06258/) (2023)
- [I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses](https://mlsanity.com/paper/2309.13550/) (2023)
- [Chest X-ray Image Classification: A Causal Perspective](https://mlsanity.com/paper/2305.12072/) (2023)
- [WeakMedSAM: Weakly-Supervised Medical Image Segmentation via SAM with Sub-Class Exploration and Prompt Affinity Mining](https://mlsanity.com/paper/2503.04106/) (2025)
- [Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image](https://mlsanity.com/paper/2406.14050/) (2024)
- [ThoraX-PriorNet: A Novel Attention-Based Architecture Using Anatomical Prior Probability Maps for Thoracic Disease Classification](https://mlsanity.com/paper/2210.02998/) (2022)
- [Expert Uncertainty and Severity Aware Chest X-Ray Classification by Multi-Relationship Graph Learning](https://mlsanity.com/paper/2309.03331/) (2023)
- [Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation](https://mlsanity.com/paper/2411.04607/) (2024)
- [Deep Reinforcement Learning Framework for Thoracic Diseases Classification via Prior Knowledge Guidance](https://mlsanity.com/paper/2306.01232/) (2023)
- [DisAsymNet: Disentanglement of Asymmetrical Abnormality on Bilateral Mammograms using Self-adversarial Learning](https://mlsanity.com/paper/2307.02935/) (2023)
- [Learning to Generalize towards Unseen Domains via a Content-Aware Style Invariant Model for Disease Detection from Chest X-rays](https://mlsanity.com/paper/2302.13991/) (2023)
- [Instrumental Variable Learning for Chest X-ray Classification](https://mlsanity.com/paper/2305.12070/) (2023)
- [Faster ISNet for Background Bias Mitigation on Deep Neural Networks](https://mlsanity.com/paper/2401.08409/) (2024)
- [CLARiTy: A Vision Transformer for Multi-Label Classification and Weakly-Supervised Localization of Chest X-ray Pathologies](https://mlsanity.com/paper/2512.16700/) (2025)
- [Can Agents Distinguish Visually Hard-to-Separate Diseases in a Zero-Shot Setting? A Pilot Study](https://mlsanity.com/paper/2602.22959/) (2026)
- [Rethinking Saliency Map: An Context-aware Perturbation Method to Explain EEG-based Deep Learning Model](https://mlsanity.com/paper/2205.14976/) (2022)
- [Applications of Sequential Learning for Medical Image Classification](https://mlsanity.com/paper/2309.14591/) (2023)
- [Artificially Generated Visual Scanpath Improves Multi-label Thoracic Disease Classification in Chest X-Ray Images](https://mlsanity.com/paper/2503.00657/) (2025)
- [A Hierarchical Slice Attention Network for Appendicitis Classification in 3D CT Scans](https://mlsanity.com/paper/2506.23209/) (2025)
- [Inhibited Self-Attention: Sharpening Focus in Vision Transformers](https://mlsanity.com/paper/2607.12881/) (2026)
- [Exponential Pixelating Integral transform with dual fractal features for enhanced chest X-ray abnormality detection](https://mlsanity.com/paper/2609.10988/) (2026)

## Abstract (arXiv)

We consider the problem of abnormality localization for clinical applications. While deep learning has driven much recent progress in medical imaging, many clinical challenges are not fully addressed, limiting its broader usage. While recent methods report high diagnostic accuracies, physicians have concerns trusting these algorithm results for diagnostic decision-making purposes because of a general lack of algorithm decision reasoning and interpretability. One potential way to address this problem is to further train these models to localize abnormalities in addition to just classifying them. However, doing this accurately will require a large amount of disease localization annotations by clinical experts, a task that is prohibitively expensive to accomplish for most applications. In this work, we take a step towards addressing these issues by means of a new attention-driven weakly supervised algorithm comprising a hierarchical attention mining framework that unifies activation- and gradient-based visual attention in a holistic manner. Our key algorithmic innovations include the design of explicit ordinal attention constraints, enabling principled model training in a weakly-supervised fashion, while also facilitating the generation of visual-attention-driven model explanations by means of localization cues. On two large-scale chest X-ray datasets (NIH ChestX-ray14 and CheXpert), we demonstrate significant localization performance improvements over the current state of the art while also achieving competitive classification performance. Our code is available on https://github.com/oyxhust/HAM.