# ConvFormer: Combining CNN and Transformer for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2211.08564 · HTML: https://mlsanity.com/paper/2211.08564/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation](https://mlsanity.com/paper/2004.08790/) (2020)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.03024/) (2021)
- [Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network](https://mlsanity.com/paper/1703.02719/) (2017)
- [Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation](https://mlsanity.com/paper/2002.02255/) (2020)
- [MISSFormer: An Effective Medical Image Segmentation Transformer](https://mlsanity.com/paper/2109.07162/) (2021)
- [Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss](https://mlsanity.com/paper/1804.10916/) (2018)
- [Star Shape Prior in Fully Convolutional Networks for Skin Lesion Segmentation](https://mlsanity.com/paper/1806.08437/) (2018)
- [Improving Dermoscopic Image Segmentation with Enhanced Convolutional-Deconvolutional Networks](https://mlsanity.com/paper/1709.09780/) (2017)
- [Decompose-and-Integrate Learning for Multi-class Segmentation in Medical Images](https://mlsanity.com/paper/1906.02901/) (2019)

## Built on by (most cited first, 10 of 10)

- [PixMamba: Leveraging State Space Models in a Dual-Level Architecture for Underwater Image Enhancement](https://mlsanity.com/paper/2406.08444/) (2024)
- [Sli2Vol+: Segmenting 3D Medical Images Based on an Object Estimation Guided Correspondence Flow Network](https://mlsanity.com/paper/2411.13873/) (2024)
- [Topo-VM-UNetV2: Encoding Topology into Vision Mamba UNet for Polyp Segmentation](https://mlsanity.com/paper/2505.06210/) (2025)
- [Hierarchical LoG Bayesian Neural Network for Enhanced Aorta Segmentation](https://mlsanity.com/paper/2501.10615/) (2025)
- [Imitating Radiological Scrolling: A Global-Local Attention Model for 3D Chest CT Volumes Multi-Label Anomaly Classification](https://mlsanity.com/paper/2503.20652/) (2025)
- [Improving Prostate Gland Segmentation Using Transformer based Architectures](https://mlsanity.com/paper/2506.14844/) (2025)
- [JDPNet: A Network Based on Joint Degradation Processing for Underwater Image Enhancement](https://mlsanity.com/paper/2512.20213/) (2025)
- [ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching](https://mlsanity.com/paper/2604.25065/) (2026)
- [Explainable Flood Segmentation on Sentinel-1 SAR1 Imagery Using CNN and Transformer Architectures](https://mlsanity.com/paper/2606.16302/) (2026)
- [LightMIS: Ultra-Lightweight Medical Image Segmentation Without a Stage-Wise Decoder](https://mlsanity.com/paper/2609.28327/) (2026)

## Abstract (arXiv)

Convolutional neural network (CNN) based methods have achieved great successes in medical image segmentation, but their capability to learn global representations is still limited due to using small effective receptive fields of convolution operations. Transformer based methods are capable of modelling long-range dependencies of information for capturing global representations, yet their ability to model local context is lacking. Integrating CNN and Transformer to learn both local and global representations while exploring multi-scale features is instrumental in further improving medical image segmentation. In this paper, we propose a hierarchical CNN and Transformer hybrid architecture, called ConvFormer, for medical image segmentation. ConvFormer is based on several simple yet effective designs. (1) A feed forward module of Deformable Transformer (DeTrans) is re-designed to introduce local information, called Enhanced DeTrans. (2) A residual-shaped hybrid stem based on a combination of convolutions and Enhanced DeTrans is developed to capture both local and global representations to enhance representation ability. (3) Our encoder utilizes the residual-shaped hybrid stem in a hierarchical manner to generate feature maps in different scales, and an additional Enhanced DeTrans encoder with residual connections is built to exploit multi-scale features with feature maps of different scales as input. Experiments on several datasets show that our ConvFormer, trained from scratch, outperforms various CNN- or Transformer-based architectures, achieving state-of-the-art performance.