# Towards Optimal Patch Size in Vision Transformers for Tumor Segmentation

arXiv: https://arxiv.org/abs/2308.16598 · HTML: https://mlsanity.com/paper/2308.16598/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 12 of 12)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation](https://mlsanity.com/paper/2107.03442/) (2021)
- [High Tissue Contrast MRI Synthesis Using Multi-Stage Attention-GAN for Glioma Segmentation](https://mlsanity.com/paper/2006.05030/) (2020)
- [Convolutional 3D to 2D Patch Conversion for Pixel-wise Glioma Segmentation in MRI Scans](https://mlsanity.com/paper/2010.10612/) (2020)

## Built on by (most cited first, 4 of 4)

- [Finding Reproducible and Prognostic Radiomic Features in Variable Slice Thickness Contrast Enhanced CT of Colorectal Liver Metastases](https://mlsanity.com/paper/2501.11221/) (2025)
- [Masked Autoencoders for Ultrasound Signals: Robust Representation Learning for Downstream Applications](https://mlsanity.com/paper/2508.20622/) (2025)
- [Spectral Adapters for Segment Anything Model-based Segmentation of Colorectal Liver Metastases in Computed Tomography](https://mlsanity.com/paper/2609.11703/) (2026)
- [Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT](https://mlsanity.com/paper/2609.14106/) (2026)

## Abstract (arXiv)

Detection of tumors in metastatic colorectal cancer (mCRC) plays an essential role in the early diagnosis and treatment of liver cancer. Deep learning models backboned by fully convolutional neural networks (FCNNs) have become the dominant model for segmenting 3D computerized tomography (CT) scans. However, since their convolution layers suffer from limited kernel size, they are not able to capture long-range dependencies and global context. To tackle this restriction, vision transformers have been introduced to solve FCNN's locality of receptive fields. Although transformers can capture long-range features, their segmentation performance decreases with various tumor sizes due to the model sensitivity to the input patch size. While finding an optimal patch size improves the performance of vision transformer-based models on segmentation tasks, it is a time-consuming and challenging procedure. This paper proposes a technique to select the vision transformer's optimal input multi-resolution image patch size based on the average volume size of metastasis lesions. We further validated our suggested framework using a transfer-learning technique, demonstrating that the highest Dice similarity coefficient (DSC) performance was obtained by pre-training on training data with a larger tumour volume using the suggested ideal patch size and then training with a smaller one. We experimentally evaluate this idea through pre-training our model on a multi-resolution public dataset. Our model showed consistent and improved results when applied to our private multi-resolution mCRC dataset with a smaller average tumor volume. This study lays the groundwork for optimizing semantic segmentation of small objects using vision transformers. The implementation source code is available at:https://github.com/Ramtin-Mojtahedi/OVTPS.