# Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

arXiv: https://arxiv.org/abs/2211.11943 · HTML: https://mlsanity.com/paper/2211.11943/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 96 of its 112 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 30 of 96)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)

## Built on by (most cited first, 30 of 81)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [Large Selective Kernel Network for Remote Sensing Object Detection](https://mlsanity.com/paper/2303.09030/) (2023)
- [InceptionNeXt: When Inception Meets ConvNeXt](https://mlsanity.com/paper/2303.16900/) (2023)
- [RMT: Retentive Networks Meet Vision Transformers](https://mlsanity.com/paper/2309.11523/) (2023)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [Scale-Aware Modulation Meet Transformer](https://mlsanity.com/paper/2307.08579/) (2023)
- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection](https://mlsanity.com/paper/2308.05480/) (2023)
- [Breaking the Low-Rank Dilemma of Linear Attention](https://mlsanity.com/paper/2411.07635/) (2024)
- [DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation](https://mlsanity.com/paper/2504.04701/) (2025)
- [Lightweight Vision Transformer with Bidirectional Interaction](https://mlsanity.com/paper/2306.00396/) (2023)
- [OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels](https://mlsanity.com/paper/2502.20087/) (2025)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [See More Details: Efficient Image Super-Resolution by Experts Mining](https://mlsanity.com/paper/2402.03412/) (2024)
- [TransXNet: Learning Both Global and Local Dynamics with a Dual Dynamic Token Mixer for Visual Recognition](https://mlsanity.com/paper/2310.19380/) (2023)
- [DefMamba: Deformable Visual State Space Model](https://mlsanity.com/paper/2504.05794/) (2025)
- [PMAA: A Progressive Multi-scale Attention Autoencoder Model for High-performance Cloud Removal from Multi-temporal Satellite Imagery](https://mlsanity.com/paper/2303.16565/) (2023)
- [Efficient Modulation for Vision Networks](https://mlsanity.com/paper/2403.19963/) (2024)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [ADNet: Lane Shape Prediction via Anchor Decomposition](https://mlsanity.com/paper/2308.10481/) (2023)
- [SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks](https://mlsanity.com/paper/2409.09649/) (2024)
- [Strip R-CNN: Large Strip Convolution for Remote Sensing Object Detection](https://mlsanity.com/paper/2501.03775/) (2025)
- [Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach](https://mlsanity.com/paper/2401.05633/) (2024)
- [ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attention](https://mlsanity.com/paper/2405.18425/) (2024)
- [Recaptured Raw Screen Image and Video Demoir'eing via Channel and Spatial Modulations](https://mlsanity.com/paper/2310.20332/) (2023)
- [Efficient Degradation-aware Any Image Restoration](https://mlsanity.com/paper/2405.15475/) (2024)
- [FreCT: Frequency-augmented Convolutional Transformer for Robust Time Series Anomaly Detection](https://mlsanity.com/paper/2505.00941/) (2025)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)

## Abstract (arXiv)

This paper does not attempt to design a state-of-the-art method for visual recognition but investigates a more efficient way to make use of convolutions to encode spatial features. By comparing the design principles of the recent convolutional neural networks ConvNets) and Vision Transformers, we propose to simplify the self-attention by leveraging a convolutional modulation operation. We show that such a simple approach can better take advantage of the large kernels (>=7x7) nested in convolutional layers. We build a family of hierarchical ConvNets using the proposed convolutional modulation, termed Conv2Former. Our network is simple and easy to follow. Experiments show that our Conv2Former outperforms existent popular ConvNets and vision Transformers, like Swin Transformer and ConvNeXt in all ImageNet classification, COCO object detection and ADE20k semantic segmentation.