# Multi-task UNet: Jointly Boosting Saliency Prediction and Disease Classification on Chest X-ray Images

arXiv: https://arxiv.org/abs/2202.07118 · HTML: https://mlsanity.com/paper/2202.07118/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 65 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [Auxiliary Tasks in Multi-task Learning](https://mlsanity.com/paper/1805.06334/) (2018)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)

## Built on by (most cited first, 4 of 4)

- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses](https://mlsanity.com/paper/2309.13550/) (2023)
- [Weakly Supervised Intracranial Aneurysm Detection and Segmentation in MR angiography via Multi-task UNet with Vesselness Prior](https://mlsanity.com/paper/2508.00235/) (2025)
- [Fully Automatic Data Labeling for Ultrasound Screen Detection](https://mlsanity.com/paper/2511.13197/) (2025)

## Abstract (arXiv)

Human visual attention has recently shown its distinct capability in boosting machine learning models. However, studies that aim to facilitate medical tasks with human visual attention are still scarce. To support the use of visual attention, this paper describes a novel deep learning model for visual saliency prediction on chest X-ray (CXR) images. To cope with data deficiency, we exploit the multi-task learning method and tackles disease classification on CXR simultaneously. For a more robust training process, we propose a further optimized multi-task learning scheme to better handle model overfitting. Experiments show our proposed deep learning model with our new learning scheme can outperform existing methods dedicated either for saliency prediction or image classification. The code used in this paper is available at https://github.com/hz-zhu/MT-UNet.