# S2-MLPv2: Improved Spatial-Shift MLP Architecture for Vision

arXiv: https://arxiv.org/abs/2108.01072 · HTML: https://mlsanity.com/paper/2108.01072/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 2 Aug 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Global Filter Networks for Image Classification](https://mlsanity.com/paper/2107.00645/) (2021)

## Built on by (most cited first, 24 of 24)

- [Focal Modulation Networks](https://mlsanity.com/paper/2203.11926/) (2022)
- [PointMixer: MLP-Mixer for Point Cloud Understanding](https://mlsanity.com/paper/2111.11187/) (2021)
- [Hire-MLP: Vision MLP via Hierarchical Rearrangement](https://mlsanity.com/paper/2108.13341/) (2021)
- [Are we ready for a new paradigm shift? A Survey on Visual Deep MLP](https://mlsanity.com/paper/2111.04060/) (2021)
- [BALF: Simple and Efficient Blur Aware Local Feature Detector](https://mlsanity.com/paper/2211.14731/) (2022)
- [DynaMixer: A Vision MLP Architecture with Dynamic Mixing](https://mlsanity.com/paper/2201.12083/) (2022)
- [3D Medical Point Transformer: Introducing Convolution to Attention Networks for Medical Point Cloud Analysis](https://mlsanity.com/paper/2112.04863/) (2021)
- [Sequencer: Deep LSTM for Image Classification](https://mlsanity.com/paper/2205.01972/) (2022)
- [Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs](https://mlsanity.com/paper/2203.03990/) (2022)
- [BOAT: Bilateral Local Attention Vision Transformer](https://mlsanity.com/paper/2201.13027/) (2022)
- [Multilinear Operator Networks](https://mlsanity.com/paper/2401.17992/) (2024)
- [MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing](https://mlsanity.com/paper/2206.06292/) (2022)
- [CrowdMLP: Weakly-Supervised Crowd Counting via Multi-Granularity MLP](https://mlsanity.com/paper/2203.08219/) (2022)
- [RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?](https://mlsanity.com/paper/2108.04384/) (2021)
- [Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation](https://mlsanity.com/paper/2305.17644/) (2023)
- [MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition](https://mlsanity.com/paper/2202.08456/) (2022)
- [R2-MLP: Round-Roll MLP for Multi-View 3D Object Recognition](https://mlsanity.com/paper/2211.11085/) (2022)
- [Efficient Attention-free Video Shift Transformers](https://mlsanity.com/paper/2208.11108/) (2022)
- [TriMLP: Revenge of a MLP-like Architecture in Sequential Recommendation](https://mlsanity.com/paper/2305.14675/) (2023)
- [LSU-Net: Lightweight Automatic Organs Segmentation Network For Medical Images](https://mlsanity.com/paper/2502.00042/) (2025)
- [Dynamic Spectrum Mixer for Visual Recognition](https://mlsanity.com/paper/2309.06721/) (2023)
- [Dimension Mixer: Group Mixing of Input Dimensions for Efficient Function Approximation](https://mlsanity.com/paper/2311.18735/) (2023)
- [Automated Neural Architecture Design for Industrial Defect Detection](https://mlsanity.com/paper/2510.06669/) (2025)
- [Parameter-Efficient 3D Segmentation of Liver and Liver tumors: Depthwise factorization Scales Better Than Dense Convolution with Spatial Dimensionality](https://mlsanity.com/paper/2609.33077/) (2026)

## Abstract (arXiv)

Recently, MLP-based vision backbones emerge. MLP-based vision architectures with less inductive bias achieve competitive performance in image recognition compared with CNNs and vision Transformers. Among them, spatial-shift MLP (S2-MLP), adopting the straightforward spatial-shift operation, achieves better performance than the pioneering works including MLP-mixer and ResMLP. More recently, using smaller patches with a pyramid structure, Vision Permutator (ViP) and Global Filter Network (GFNet) achieve better performance than S2-MLP. In this paper, we improve the S2-MLP vision backbone. We expand the feature map along the channel dimension and split the expanded feature map into several parts. We conduct different spatial-shift operations on split parts. Meanwhile, we exploit the split-attention operation to fuse these split parts. Moreover, like the counterparts, we adopt smaller-scale patches and use a pyramid structure for boosting the image recognition accuracy. We term the improved spatial-shift MLP vision backbone as S2-MLPv2. Using 55M parameters, our medium-scale model, S2-MLPv2-Medium achieves an 83.6% top-1 accuracy on the ImageNet-1K benchmark using 224× 224 images without self-attention and external training data.