# MT-TransUNet: Mediating Multi-Task Tokens in Transformers for Skin Lesion Segmentation and Classification

arXiv: https://arxiv.org/abs/2112.01767 · HTML: https://mlsanity.com/paper/2112.01767/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Semi-supervised Medical Image Segmentation through Dual-task Consistency](https://mlsanity.com/paper/2009.04448/) (2020)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images](https://mlsanity.com/paper/1806.01313/) (2018)
- [A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification](https://mlsanity.com/paper/1903.03313/) (2019)
- [Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation](https://mlsanity.com/paper/1806.08437/) (2018)
- [Image Classification of Melanoma, Nevus and Seborrheic Keratosis by Deep Neural Network Ensemble](https://mlsanity.com/paper/1703.03108/) (2017)
- [Improving Dermoscopic Image Segmentation with Enhanced Convolutional-Deconvolutional Networks](https://mlsanity.com/paper/1709.09780/) (2017)
- [A Novel Multi-task Deep Learning Model for Skin Lesion Segmentation and Classification](https://mlsanity.com/paper/1703.01025/) (2017)
- [SLSDeep: Skin Lesion Segmentation Based on Dilated Residual and Pyramid Pooling Networks](https://mlsanity.com/paper/1805.10241/) (2018)
- [Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions](https://mlsanity.com/paper/1703.01976/) (2017)
- [Automatic Skin Lesion Analysis using Large-scale Dermoscopy Images and Deep Residual Networks](https://mlsanity.com/paper/1703.04197/) (2017)
- [RECOD Titans at ISIC Challenge 2017](https://mlsanity.com/paper/1703.04819/) (2017)
- [Automatic skin lesion segmentation on dermoscopic images by the means of superpixel merging](https://mlsanity.com/paper/1808.06759/) (2018)
- [Dense Pooling layers in Fully Convolutional Network for Skin Lesion Segmentation](https://mlsanity.com/paper/1712.10207/) (2017)

## Built on by (most cited first, 7 of 7)

- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data](https://mlsanity.com/paper/2510.14831/) (2025)
- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)
- [GS-TransUNet: Integrated 2D Gaussian Splatting and Transformer UNet for Accurate Skin Lesion Analysis](https://mlsanity.com/paper/2502.16748/) (2025)
- [JOINEDTrans: Prior Guided Multi-task Transformer for Joint Optic Disc/Cup Segmentation and Fovea Detection](https://mlsanity.com/paper/2305.11504/) (2023)
- [MAPUNetR: A Hybrid Vision Transformer and U-Net Architecture for Efficient and Interpretable Medical Image Segmentation](https://mlsanity.com/paper/2410.22223/) (2024)
- [IViT: A Novel Interpretable Visual Transformer for Skin Disease Detection](https://mlsanity.com/paper/2606.22892/) (2026)

## Abstract (arXiv)

Recent advances in automated skin cancer diagnosis have yielded performance on par with board-certified dermatologists. However, these approaches formulated skin cancer diagnosis as a simple classification task, dismissing the potential benefit from lesion segmentation. We argue that an accurate lesion segmentation can supplement the classification task with additive lesion information, such as asymmetry, border, intensity, and physical size; in turn, a faithful lesion classification can support the segmentation task with discriminant lesion features. To this end, this paper proposes a new multi-task framework, named MT-TransUNet, which is capable of segmenting and classifying skin lesions collaboratively by mediating multi-task tokens in Transformers. Furthermore, we have introduced dual-task and attended region consistency losses to take advantage of those images without pixel-level annotation, ensuring the model's robustness when it encounters the same image with an account of augmentation. Our MT-TransUNet exceeds the previous state of the art for lesion segmentation and classification tasks in ISIC-2017 and PH2; more importantly, it preserves compelling computational efficiency regarding model parameters (48M~vs.~130M) and inference speed (0.17s~vs.~2.02s per image). Code will be available at https://github.com/JingyeChen/MT-TransUNet.