# Boundary-aware Transformers for Skin Lesion Segmentation

arXiv: https://arxiv.org/abs/2110.03864 · HTML: https://mlsanity.com/paper/2110.03864/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Dilated Residual Networks](https://mlsanity.com/paper/1705.09914/) (2017)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Attention-Based Transformers for Instance Segmentation of Cells in Microstructures](https://mlsanity.com/paper/2011.09763/) (2020)

## Built on by (most cited first, 30 of 39)

- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [LViT: Language meets Vision Transformer in Medical Image Segmentation](https://mlsanity.com/paper/2206.14718/) (2022)
- [MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine](https://mlsanity.com/paper/2408.02900/) (2024)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [The Fully Convolutional Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2206.00566/) (2022)
- [A Recent Survey of Vision Transformers for Medical Image Segmentation](https://mlsanity.com/paper/2312.00634/) (2023)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [A survey, review, and future trends of skin lesion segmentation and classification](https://mlsanity.com/paper/2208.12232/) (2022)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis](https://mlsanity.com/paper/2211.11393/) (2022)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [DermoSegDiff: A Boundary-aware Segmentation Diffusion Model for Skin Lesion Delineation](https://mlsanity.com/paper/2308.02959/) (2023)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [MDViT: Multi-domain Vision Transformer for Small Medical Image Segmentation Datasets](https://mlsanity.com/paper/2307.02100/) (2023)
- [Narrowing the semantic gaps in U-Net with learnable skip connections: The case of medical image segmentation](https://mlsanity.com/paper/2312.15182/) (2023)
- [Channel prior convolutional attention for medical image segmentation](https://mlsanity.com/paper/2306.05196/) (2023)
- [XBound-Former: Toward Cross-scale Boundary Modeling in Transformers](https://mlsanity.com/paper/2206.00806/) (2022)
- [UNet-2022: Exploring Dynamics in Non-isomorphic Architecture](https://mlsanity.com/paper/2210.15566/) (2022)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [Revisiting the Shape-Bias of Deep Learning for Dermoscopic Skin Lesion Classification](https://mlsanity.com/paper/2206.06466/) (2022)
- [HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2412.11458/) (2024)
- [Memory transformers for full context and high-resolution 3D Medical Segmentation](https://mlsanity.com/paper/2210.05313/) (2022)
- [AViT: Adapting Vision Transformers for Small Skin Lesion Segmentation Datasets](https://mlsanity.com/paper/2307.13897/) (2023)
- [SkinFormer: Learning Statistical Texture Representation with Transformer for Skin Lesion Segmentation](https://mlsanity.com/paper/2409.08652/) (2024)
- [Global-Local Medical SAM Adaptor Based on Full Adaption](https://mlsanity.com/paper/2409.17486/) (2024)
- [SAM-Based Building Change Detection with Distribution-Aware Fourier Adaptation and Edge-Constrained Warping](https://mlsanity.com/paper/2504.12619/) (2025)

## Abstract (arXiv)

Skin lesion segmentation from dermoscopy images is of great importance for improving the quantitative analysis of skin cancer. However, the automatic segmentation of melanoma is a very challenging task owing to the large variation of melanoma and ambiguous boundaries of lesion areas. While convolutional neutral networks (CNNs) have achieved remarkable progress in this task, most of existing solutions are still incapable of effectively capturing global dependencies to counteract the inductive bias caused by limited receptive fields. Recently, transformers have been proposed as a promising tool for global context modeling by employing a powerful global attention mechanism, but one of their main shortcomings when applied to segmentation tasks is that they cannot effectively extract sufficient local details to tackle ambiguous boundaries. We propose a novel boundary-aware transformer (BAT) to comprehensively address the challenges of automatic skin lesion segmentation. Specifically, we integrate a new boundary-wise attention gate (BAG) into transformers to enable the whole network to not only effectively model global long-range dependencies via transformers but also, simultaneously, capture more local details by making full use of boundary-wise prior knowledge. Particularly, the auxiliary supervision of BAG is capable of assisting transformers to learn position embedding as it provides much spatial information. We conducted extensive experiments to evaluate the proposed BAT and experiments corroborate its effectiveness, consistently outperforming state-of-the-art methods in two famous datasets.