# InceptionNeXt: When Inception Meets ConvNeXt

arXiv: https://arxiv.org/abs/2303.16900 · HTML: https://mlsanity.com/paper/2303.16900/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 93 references that are in the map.
- First posted on arXiv 29 Mar 2023.

## Builds on (most cited first, 30 of 73)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)

## Built on by (most cited first, 30 of 86)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [RMT: Retentive Networks Meet Vision Transformers](https://mlsanity.com/paper/2309.11523/) (2023)
- [Poly Kernel Inception Network for Remote Sensing Detection](https://mlsanity.com/paper/2403.06258/) (2024)
- [reBEN: Refined BigEarthNet Dataset for Remote Sensing Image Analysis](https://mlsanity.com/paper/2407.03653/) (2024)
- [StableCodec: Taming One-Step Diffusion for Extreme Image Compression](https://mlsanity.com/paper/2506.21977/) (2025)
- [PeLK: Parameter-efficient Large Kernel ConvNets with Peripheral Convolution](https://mlsanity.com/paper/2403.07589/) (2024)
- [Context-Guided Spatial Feature Reconstruction for Efficient Semantic Segmentation](https://mlsanity.com/paper/2405.06228/) (2024)
- [OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels](https://mlsanity.com/paper/2502.20087/) (2025)
- [Dropout the High-rate Downsampling: A Novel Design Paradigm for UHD Image Restoration](https://mlsanity.com/paper/2411.06456/) (2024)
- [ViT3: Unlocking Test-Time Training in Vision](https://mlsanity.com/paper/2512.01643/) (2025)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)
- [TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mamba](https://mlsanity.com/paper/2411.17473/) (2024)
- [DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs](https://mlsanity.com/paper/2403.19588/) (2024)
- [CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications](https://mlsanity.com/paper/2408.03703/) (2024)
- [RFAConv: Receptive-Field Attention Convolution for Improving Convolutional Neural Networks](https://mlsanity.com/paper/2304.03198/) (2023)
- [SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention](https://mlsanity.com/paper/2407.05128/) (2024)
- [DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions](https://mlsanity.com/paper/2502.05091/) (2025)
- [Frequency-Dynamic Attention Modulation for Dense Prediction](https://mlsanity.com/paper/2507.12006/) (2025)
- [DAMamba: Vision State Space Model with Dynamic Adaptive Scan](https://mlsanity.com/paper/2502.12627/) (2025)
- [NTIRE 2025 Challenge on RAW Image Restoration and Super-Resolution](https://mlsanity.com/paper/2506.02197/) (2025)
- [Ultra-Low Bitrate Perceptual Image Compression with Shallow Encoder](https://mlsanity.com/paper/2512.12229/) (2025)
- [FalconNet: Factorization for the Light-weight ConvNets](https://mlsanity.com/paper/2306.06365/) (2023)
- [DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision Recognition](https://mlsanity.com/paper/2503.14867/) (2025)
- [StarLKNet: Star Mixup with Large Kernel Networks for Palm Vein Identification](https://mlsanity.com/paper/2405.12721/) (2024)
- [Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularities](https://mlsanity.com/paper/2504.01954/) (2025)
- [SSA-Seg: Semantic and Spatial Adaptive Pixel-level Classifier for Semantic Segmentation](https://mlsanity.com/paper/2405.06525/) (2024)
- [RepNeXt: A Fast Multi-Scale CNN using Structural Reparameterization](https://mlsanity.com/paper/2406.16004/) (2024)
- [MEET: A Million-Scale Dataset for Fine-Grained Geospatial Scene Classification with Zoom-Free Remote Sensing Imagery](https://mlsanity.com/paper/2503.11219/) (2025)
- [Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning](https://mlsanity.com/paper/2410.06373/) (2024)
- [S2AFormer: Strip Self-Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.22195/) (2025)

## Abstract (arXiv)

Inspired by the long-range modeling ability of ViTs, large-kernel convolutions are widely studied and adopted recently to enlarge the receptive field and improve model performance, like the remarkable work ConvNeXt which employs 7x7 depthwise convolution. Although such depthwise operator only consumes a few FLOPs, it largely harms the model efficiency on powerful computing devices due to the high memory access costs. For example, ConvNeXt-T has similar FLOPs with ResNet-50 but only achieves ~60% throughputs when trained on A100 GPUs with full precision. Although reducing the kernel size of ConvNeXt can improve speed, it results in significant performance degradation, which poses a challenging problem: How to speed up large-kernel-based CNN models while preserving their performance. To tackle this issue, inspired by Inceptions, we propose to decompose large-kernel depthwise convolution into four parallel branches along channel dimension, i.e., small square kernel, two orthogonal band kernels, and an identity mapping. With this new Inception depthwise convolution, we build a series of networks, namely IncepitonNeXt, which not only enjoy high throughputs but also maintain competitive performance. For instance, InceptionNeXt-T achieves 1.6x higher training throughputs than ConvNeX-T, as well as attains 0.2% top-1 accuracy improvement on ImageNet-1K. We anticipate InceptionNeXt can serve as an economical baseline for future architecture design to reduce carbon footprint. Code is available at https://github.com/sail-sg/inceptionnext.