# Narrowing the semantic gaps in U-Net with learnable skip connections: The case of medical image segmentation

arXiv: https://arxiv.org/abs/2312.15182 · HTML: https://mlsanity.com/paper/2312.15182/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 23 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation](https://mlsanity.com/paper/1912.05074/) (2019)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest](https://mlsanity.com/paper/1603.00275/) (2016)
- [UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer](https://mlsanity.com/paper/2109.04335/) (2021)
- [Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images](https://mlsanity.com/paper/2004.14133/) (2020)
- [Multi-Compound Transformer for Accurate Biomedical Image Segmentation](https://mlsanity.com/paper/2106.14385/) (2021)
- [Boundary-aware Transformers for Skin Lesion Segmentation](https://mlsanity.com/paper/2110.03864/) (2021)
- [Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2310.11320/) (2023)
- [A Multi-Branch Hybrid Transformer Networkfor Corneal Endothelial Cell Segmentation](https://mlsanity.com/paper/2106.07557/) (2021)
- [HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images](https://mlsanity.com/paper/2209.00729/) (2022)
- [Select, Attend, and Transfer: Light, Learnable Skip Connections](https://mlsanity.com/paper/1804.05181/) (2018)
- [UNet#: A UNet-like Redesigning Skip Connections for Medical Image Segmentation](https://mlsanity.com/paper/2205.11759/) (2022)

## Built on by (most cited first, 9 of 9)

- [Residual-SwinCA-Net: A Channel-Aware Integrated Residual CNN-Swin Transformer for Malignant Lesion Segmentation in BUSI](https://mlsanity.com/paper/2512.08243/) (2025)
- [FIPGNet:Pyramid grafting network with feature interaction strategies](https://mlsanity.com/paper/2407.04085/) (2024)
- [InceptionMamba: Efficient Multi-Stage Feature Enhancement with Selective State Space Model for Microscopic Medical Image Segmentation](https://mlsanity.com/paper/2506.12208/) (2025)
- [MACMD: Multi-dilated Contextual Attention and Channel Mixer Decoding for Medical Image Segmentation](https://mlsanity.com/paper/2511.05803/) (2025)
- [XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation](https://mlsanity.com/paper/2604.03297/) (2026)
- [Mtextsuperscript4Fuse: Lightweight State-Space MoE with a Cross-Scale Gating Bridge for Brain Tumor Segmentation](https://mlsanity.com/paper/2605.02444/) (2026)
- [MedCAGD: Context-Aware Gated Decoder for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2607.00409/) (2026)
- [Towards Investigating Residual Hearing Loss: Quantification of Fibrosis in a Novel Cochlear OCT Dataset](https://mlsanity.com/paper/2608.21189/) (2026)
- [RFS-UNet: Decoder-Conditioned High-Resolution Skip Recalibration for Bone-Selective DRR Synthesis](https://mlsanity.com/paper/2609.08044/) (2026)

## Abstract (arXiv)

Most state-of-the-art methods for medical image segmentation adopt the encoder-decoder architecture. However, this U-shaped framework still has limitations in capturing the non-local multi-scale information with a simple skip connection. To solve the problem, we firstly explore the potential weakness of skip connections in U-Net on multiple segmentation tasks, and find that i) not all skip connections are useful, each skip connection has different contribution; ii) the optimal combinations of skip connections are different, relying on the specific datasets. Based on our findings, we propose a new segmentation framework, named UDTransNet, to solve three semantic gaps in U-Net. Specifically, we propose a Dual Attention Transformer (DAT) module for capturing the channel- and spatial-wise relationships to better fuse the encoder features, and a Decoder-guided Recalibration Attention (DRA) module for effectively connecting the DAT tokens and the decoder features to eliminate the inconsistency. Hence, both modules establish a learnable connection to solve the semantic gaps between the encoder and the decoder, which leads to a high-performance segmentation model for medical images. Comprehensive experimental results indicate that our UDTransNet produces higher evaluation scores and finer segmentation results with relatively fewer parameters over the state-of-the-art segmentation methods on different public datasets. Code: https://github.com/McGregorWwww/UDTransNet.