# HF-UNet: Learning Hierarchically Inter-Task Relevance in Multi-Task U-Net for Accurate Prostate Segmentation

arXiv: https://arxiv.org/abs/2005.10439 · HTML: https://mlsanity.com/paper/2005.10439/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 8 of 8)

- [A Deep Learning-based Quality Assessment and Segmentation System with a Large-scale Benchmark Dataset for Optical Coherence Tomographic Angiography Image](https://mlsanity.com/paper/2107.10476/) (2021)
- [FocalUNETR: A Focal Transformer for Boundary-aware Segmentation of CT Images](https://mlsanity.com/paper/2210.03189/) (2022)
- [Hierarchical Information-sharing Convolutional Neural Network for the Prediction of Arctic Sea Ice Concentration and Velocity](https://mlsanity.com/paper/2311.00167/) (2023)
- [Meta-Analysis of Transfer Learning for Segmentation of Brain Lesions](https://mlsanity.com/paper/2306.11714/) (2023)
- [Hybrid Dual Mean-Teacher Network With Double-Uncertainty Guidance for Semi-Supervised Segmentation of MRI Scans](https://mlsanity.com/paper/2303.05126/) (2023)
- [Detection-Guided Deep Learning-Based Model with Spatial Regularization for Lung Nodule Segmentation](https://mlsanity.com/paper/2410.20154/) (2024)
- [Multi-Task Cooperative Learning via Searching for Flat Minima](https://mlsanity.com/paper/2309.12090/) (2023)
- [MT-CYP-Net: Multi-Task Network for Pixel-Level Crop Yield Prediction Under Very Few Samples](https://mlsanity.com/paper/2505.12069/) (2025)

## Abstract (arXiv)

Accurate segmentation of the prostate is a key step in external beam radiation therapy treatments. In this paper, we tackle the challenging task of prostate segmentation in CT images by a two-stage network with 1) the first stage to fast localize, and 2) the second stage to accurately segment the prostate. To precisely segment the prostate in the second stage, we formulate prostate segmentation into a multi-task learning framework, which includes a main task to segment the prostate, and an auxiliary task to delineate the prostate boundary. Here, the second task is applied to provide additional guidance of unclear prostate boundary in CT images. Besides, the conventional multi-task deep networks typically share most of the parameters (i.e., feature representations) across all tasks, which may limit their data fitting ability, as the specificities of different tasks are inevitably ignored. By contrast, we solve them by a hierarchically-fused U-Net structure, namely HF-UNet. The HF-UNet has two complementary branches for two tasks, with the novel proposed attention-based task consistency learning block to communicate at each level between the two decoding branches. Therefore, HF-UNet endows the ability to learn hierarchically the shared representations for different tasks, and preserve the specificities of learned representations for different tasks simultaneously. We did extensive evaluations of the proposed method on a large planning CT image dataset, including images acquired from 339 patients. The experimental results show HF-UNet outperforms the conventional multi-task network architectures and the state-of-the-art methods.