# TransNuSeg: A Lightweight Multi-Task Transformer for Nuclei Segmentation

arXiv: https://arxiv.org/abs/2307.08051 · HTML: https://mlsanity.com/paper/2307.08051/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNeXt: MLP-based Rapid Medical Image Segmentation Network](https://mlsanity.com/paper/2203.04967/) (2022)
- [DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation](https://mlsanity.com/paper/1604.02677/) (2016)
- [CIA-Net: Robust Nuclei Instance Segmentation with Contour-aware Information Aggregation](https://mlsanity.com/paper/1903.05358/) (2019)
- [HistoSeg : Quick attention with multi-loss function for multi-structure segmentation in digital histology images](https://mlsanity.com/paper/2209.00729/) (2022)
- [Federated Learning for Chronic Obstructive Pulmonary Disease Classification with Partial Personalized Attention Mechanism](https://mlsanity.com/paper/2210.16142/) (2022)

## Built on by (most cited first, 15 of 15)

- [FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images](https://mlsanity.com/paper/2403.09827/) (2024)
- [Co-synthesis of Histopathology Nuclei Image-Label Pairs using a Context-Conditioned Joint Diffusion Model](https://mlsanity.com/paper/2407.14434/) (2024)
- [SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery](https://mlsanity.com/paper/2510.18740/) (2025)
- [Category Prompt Mamba Network for Nuclei Segmentation and Classification](https://mlsanity.com/paper/2503.10422/) (2025)
- [The Four Color Theorem for Cell Instance Segmentation](https://mlsanity.com/paper/2506.09724/) (2025)
- [Spatial-Frequency Dual Progressive Attention Network For Medical Image Segmentation](https://mlsanity.com/paper/2406.07952/) (2024)
- [SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation](https://mlsanity.com/paper/2508.02464/) (2025)
- [Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images](https://mlsanity.com/paper/2401.15638/) (2024)
- [A Survey on Cell Nuclei Instance Segmentation and Classification: Leveraging Context and Attention](https://mlsanity.com/paper/2407.18673/) (2024)
- [Memorizing SAM: 3D Medical Segment Anything Model with Memorizing Transformer](https://mlsanity.com/paper/2412.13908/) (2024)
- [CerraData-4MM: A multimodal benchmark dataset on Cerrado for land use and land cover classification](https://mlsanity.com/paper/2502.00083/) (2025)
- [Prompt Group-Aware Training for Robust Text-Guided Nuclei Segmentation](https://mlsanity.com/paper/2603.06384/) (2026)
- [APRIL-MedSeg: A Modular Medical Image Segmentation Toolbox Embracing Modern Paradigms](https://mlsanity.com/paper/2606.30577/) (2026)
- [Generalized Biomedicine Discovery](https://mlsanity.com/paper/2610.00120/) (2026)
- [GDBA-Net: A Gated Dual-Branch Adaptation of SAM2 for Robust Cross-Domain Nuclei Segmentation](https://mlsanity.com/paper/2610.20073/) (2026)

## Abstract (arXiv)

Nuclei appear small in size, yet, in real clinical practice, the global spatial information and correlation of the color or brightness contrast between nuclei and background, have been considered a crucial component for accurate nuclei segmentation. However, the field of automatic nuclei segmentation is dominated by Convolutional Neural Networks (CNNs), meanwhile, the potential of the recently prevalent Transformers has not been fully explored, which is powerful in capturing local-global correlations. To this end, we make the first attempt at a pure Transformer framework for nuclei segmentation, called TransNuSeg. Different from prior work, we decouple the challenging nuclei segmentation task into an intrinsic multi-task learning task, where a tri-decoder structure is employed for nuclei instance, nuclei edge, and clustered edge segmentation respectively. To eliminate the divergent predictions from different branches in previous work, a novel self distillation loss is introduced to explicitly impose consistency regulation between branches. Moreover, to formulate the high correlation between branches and also reduce the number of parameters, an efficient attention sharing scheme is proposed by partially sharing the self-attention heads amongst the tri-decoders. Finally, a token MLP bottleneck replaces the over-parameterized Transformer bottleneck for a further reduction in model complexity. Experiments on two datasets of different modalities, including MoNuSeg have shown that our methods can outperform state-of-the-art counterparts such as CA2.5-Net by 2-3% Dice with 30% fewer parameters. In conclusion, TransNuSeg confirms the strength of Transformer in the context of nuclei segmentation, which thus can serve as an efficient solution for real clinical practice. Code is available at https://github.com/zhenqi-he/transnuseg.