# SegTransVAE: Hybrid CNN -- Transformer with Regularization for medical image segmentation

arXiv: https://arxiv.org/abs/2201.08582 · HTML: https://mlsanity.com/paper/2201.08582/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 12 references that are in the map.
- First posted on arXiv 21 Jan 2022.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [TransBTS: Multimodal Brain Tumor Segmentation Using Transformer](https://mlsanity.com/paper/2103.04430/) (2021)
- [The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes](https://mlsanity.com/paper/1904.00445/) (2019)

## Built on by (most cited first, 8 of 8)

- [CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation](https://mlsanity.com/paper/2207.07370/) (2022)
- [Recent Progress in Transformer-based Medical Image Analysis](https://mlsanity.com/paper/2208.06643/) (2022)
- [Brain Tumor Segmentation Based on Deep Learning, Attention Mechanisms, and Energy-Based Uncertainty Prediction](https://mlsanity.com/paper/2401.00587/) (2023)
- [Attention Xception UNet (AXUNet): A Novel Combination of CNN and Self-Attention for Brain Tumor Segmentation](https://mlsanity.com/paper/2503.20446/) (2025)
- [Multimodal Fusion at Three Tiers: Physics-Driven Data Generation and Vision-Language Guidance for Brain Tumor Segmentation](https://mlsanity.com/paper/2507.09966/) (2025)
- [Modality-Aware and Shift Mixer for Multi-modal Brain Tumor Segmentation](https://mlsanity.com/paper/2403.02074/) (2024)
- [RF-HiT: Rectified Flow Hierarchical Transformer for General Medical Image Segmentation](https://mlsanity.com/paper/2604.19570/) (2026)
- [MultiFlow: Vision-Driven Multimodal fMRI Encoding via Stimulus-Informed Source Flow Matching](https://mlsanity.com/paper/2610.15339/) (2026)

## Abstract (arXiv)

Current research on deep learning for medical image segmentation exposes their limitations in learning either global semantic information or local contextual information. To tackle these issues, a novel network named SegTransVAE is proposed in this paper. SegTransVAE is built upon encoder-decoder architecture, exploiting transformer with the variational autoencoder (VAE) branch to the network to reconstruct the input images jointly with segmentation. To the best of our knowledge, this is the first method combining the success of CNN, transformer, and VAE. Evaluation on various recently introduced datasets shows that SegTransVAE outperforms previous methods in Dice Score and 95%-Haudorff Distance while having comparable inference time to a simple CNN-based architecture network. The source code is available at: https://github.com/itruonghai/SegTransVAE.