# Dynamic Linear Transformer for 3D Biomedical Image Segmentation

arXiv: https://arxiv.org/abs/2206.00771 · HTML: https://mlsanity.com/paper/2206.00771/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Efficient Attention: Attention with Linear Complexities](https://mlsanity.com/paper/1812.01243/) (2018)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [Optimizing Deeper Transformers on Small Datasets](https://mlsanity.com/paper/2012.15355/) (2020)
- [Fully Automated Pancreas Segmentation with Two-stage 3D Convolutional Neural Networks](https://mlsanity.com/paper/1906.01795/) (2019)
- [Hierarchical 3D Feature Learning for Pancreas Segmentation](https://mlsanity.com/paper/2109.01667/) (2021)
- [Hypernet-Ensemble Learning of Segmentation Probability for Medical Image Segmentation with Ambiguous Labels](https://mlsanity.com/paper/2112.06693/) (2021)

## Built on by (most cited first, 10 of 10)

- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [Large Scale MRI Collection and Segmentation of Cirrhotic Liver](https://mlsanity.com/paper/2410.16296/) (2024)
- [Radiomics Boosts Deep Learning Model for IPMN Classification](https://mlsanity.com/paper/2309.05857/) (2023)
- [Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation](https://mlsanity.com/paper/2407.19284/) (2024)
- [Detection of Peri-Pancreatic Edema using Deep Learning and Radiomics Techniques](https://mlsanity.com/paper/2404.17064/) (2024)
- [Selecting the Best Optimizers for Deep Learning based Medical Image Segmentation](https://mlsanity.com/paper/2302.02289/) (2023)
- [Cyst-X: A Multi-Center MRI Benchmark and Federated Learning Framework for Malignancy-Risk Stratification of Pancreatic Cystic Neoplasm](https://mlsanity.com/paper/2507.22017/) (2025)
- [Ensemble Learning with Residual Transformer for Brain Tumor Segmentation](https://mlsanity.com/paper/2308.00128/) (2023)
- [Segmentation Quality and Volumetric Accuracy in Medical Imaging](https://mlsanity.com/paper/2404.17742/) (2024)
- [Large-Scale Multi-Center CT and MRI Segmentation of Pancreas with Deep Learning](https://mlsanity.com/paper/2405.12367/) (2024)

## Abstract (arXiv)

Transformer-based neural networks have surpassed promising performance on many biomedical image segmentation tasks due to a better global information modeling from the self-attention mechanism. However, most methods are still designed for 2D medical images while ignoring the essential 3D volume information. The main challenge for 3D transformer-based segmentation methods is the quadratic complexity introduced by the self-attention mechanism citevaswani2017attention. In this paper, we propose a novel transformer architecture for 3D medical image segmentation using an encoder-decoder style architecture with linear complexity. Furthermore, we newly introduce a dynamic token concept to further reduce the token numbers for self-attention calculation. Taking advantage of the global information modeling, we provide uncertainty maps from different hierarchy stages. We evaluate this method on multiple challenging CT pancreas segmentation datasets. Our promising results show that our novel 3D Transformer-based segmentor could provide promising highly feasible segmentation performance and accurate uncertainty quantification using single annotation. Code is available https://github.com/freshman97/LinTransUNet.