# Rethinking U-net Skip Connections for Biomedical Image Segmentation

arXiv: https://arxiv.org/abs/2402.08276 · HTML: https://mlsanity.com/paper/2402.08276/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations](https://mlsanity.com/paper/1707.03237/) (2017)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks](https://mlsanity.com/paper/1803.02579/) (2018)
- [Instance Localization for Self-supervised Detection Pretraining](https://mlsanity.com/paper/2102.08318/) (2021)
- [Mitosis domain generalization in histopathology images -- The MIDOG challenge](https://mlsanity.com/paper/2204.03742/) (2022)
- [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018)
- [Domain Generalization for Mammography Detection via Multi-style and Multi-view Contrastive Learning](https://mlsanity.com/paper/2111.10827/) (2021)
- [Pan-tumor CAnine cuTaneous Cancer Histology (CATCH) dataset](https://mlsanity.com/paper/2201.11446/) (2022)
- [First U-Net Layers Contain More Domain Specific Information Than The Last Ones](https://mlsanity.com/paper/2008.07357/) (2020)
- [Lightweight Unsupervised Domain Adaptation by Convolutional Filter Reconstruction](https://mlsanity.com/paper/1603.07234/) (2016)
- [Mind the Gap: Scanner-induced domain shifts pose challenges for representation learning in histopathology](https://mlsanity.com/paper/2211.16141/) (2022)
- [Multi-Scanner Canine Cutaneous Squamous Cell Carcinoma Histopathology Dataset](https://mlsanity.com/paper/2301.04423/) (2023)

## Built on by (most cited first, 3 of 3)

- [NEBULA: Neural Empirical Bayes Under Latent Representations for Efficient and Controllable Design of Molecular Libraries](https://mlsanity.com/paper/2407.03428/) (2024)
- [Multimodal Power Outage Prediction for Rapid Disaster Response and Resource Allocation](https://mlsanity.com/paper/2410.00017/) (2024)
- [U-net based prediction of cerebrospinal fluid distribution and ventricular reflux grading](https://mlsanity.com/paper/2410.04460/) (2024)

## Abstract (arXiv)

The U-net architecture has significantly impacted deep learning-based segmentation of medical images. Through the integration of long-range skip connections, it facilitated the preservation of high-resolution features. Out-of-distribution data can, however, substantially impede the performance of neural networks. Previous works showed that the trained network layers differ in their susceptibility to this domain shift, e.g., shallow layers are more affected than deeper layers. In this work, we investigate the implications of this observation of layer sensitivity to domain shifts of U-net-style segmentation networks. By copying features of shallow layers to corresponding decoder blocks, these bear the risk of re-introducing domain-specific information. We used a synthetic dataset to model different levels of data distribution shifts and evaluated the impact on downstream segmentation performance. We quantified the inherent domain susceptibility of each network layer, using the Hellinger distance. These experiments confirmed the higher domain susceptibility of earlier network layers. When gradually removing skip connections, a decrease in domain susceptibility of deeper layers could be observed. For downstream segmentation performance, the original U-net outperformed the variant without any skip connections. The best performance, however, was achieved when removing the uppermost skip connection - not only in the presence of domain shifts but also for in-domain test data. We validated our results on three clinical datasets - two histopathology datasets and one magnetic resonance dataset - with performance increases of up to 10% in-domain and 13% cross-domain when removing the uppermost skip connection.