# All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification

arXiv: https://arxiv.org/abs/1903.05285 · HTML: https://mlsanity.com/paper/1903.05285/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2019.

## Built on by (most cited first, 30 of 38)

- [A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift](https://mlsanity.com/paper/2206.10810/) (2022)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [Label Matching Semi-Supervised Object Detection](https://mlsanity.com/paper/2206.06608/) (2022)
- [GhostNetV2: Enhance Cheap Operation with Long-Range Attention](https://mlsanity.com/paper/2211.12905/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)
- [When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism](https://mlsanity.com/paper/2201.10801/) (2022)
- [Attention Diversification for Domain Generalization](https://mlsanity.com/paper/2210.04206/) (2022)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives](https://mlsanity.com/paper/2308.02066/) (2023)
- [Genie: Show Me the Data for Quantization](https://mlsanity.com/paper/2212.04780/) (2022)
- [GhostNets on Heterogeneous Devices via Cheap Operations](https://mlsanity.com/paper/2201.03297/) (2022)
- [MoCoViT: Mobile Convolutional Vision Transformer](https://mlsanity.com/paper/2205.12635/) (2022)
- [GhostSR: Learning Ghost Features for Efficient Image Super-Resolution](https://mlsanity.com/paper/2101.08525/) (2021)
- [Fully 1×1 Convolutional Network for Lightweight Image Super-Resolution](https://mlsanity.com/paper/2307.16140/) (2023)
- [Learning Features with Parameter-Free Layers](https://mlsanity.com/paper/2202.02777/) (2022)
- [Vision Transformer with Cross-attention by Temporal Shift for Efficient Action Recognition](https://mlsanity.com/paper/2204.00452/) (2022)
- [Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification](https://mlsanity.com/paper/2407.15155/) (2024)
- [TS-SNN: Temporal Shift Module for Spiking Neural Networks](https://mlsanity.com/paper/2505.04165/) (2025)
- [RCNet: Reverse Feature Pyramid and Cross-scale Shift Network for Object Detection](https://mlsanity.com/paper/2110.12130/) (2021)
- [Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation](https://mlsanity.com/paper/2305.17644/) (2023)
- [Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensing](https://mlsanity.com/paper/2309.06047/) (2023)
- [ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effect](https://mlsanity.com/paper/2401.12736/) (2024)
- [Improving Generalization of Batch Whitening by Convolutional Unit Optimization](https://mlsanity.com/paper/2108.10629/) (2021)
- [CNNs with Multi-Level Attention for Domain Generalization](https://mlsanity.com/paper/2304.00502/) (2023)
- [Transductive CLIP with Class-Conditional Contrastive Learning](https://mlsanity.com/paper/2206.06177/) (2022)
- [ADVISE: ADaptive Feature Relevance and VISual Explanations for Convolutional Neural Networks](https://mlsanity.com/paper/2203.01289/) (2022)
- [MetaFBP: Learning to Learn High-Order Predictor for Personalized Facial Beauty Prediction](https://mlsanity.com/paper/2311.13929/) (2023)
- [ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation](https://mlsanity.com/paper/2407.02881/) (2024)
- [Rethinking the Role of Spatial Mixing](https://mlsanity.com/paper/2503.16760/) (2025)

## Abstract (arXiv)

Shift operation is an efficient alternative over depthwise separable convolution. However, it is still bottlenecked by its implementation manner, namely memory movement. To put this direction forward, a new and novel basic component named Sparse Shift Layer (SSL) is introduced in this paper to construct efficient convolutional neural networks. In this family of architectures, the basic block is only composed by 1x1 convolutional layers with only a few shift operations applied to the intermediate feature maps. To make this idea feasible, we introduce shift operation penalty during optimization and further propose a quantization-aware shift learning method to impose the learned displacement more friendly for inference. Extensive ablation studies indicate that only a few shift operations are sufficient to provide spatial information communication. Furthermore, to maximize the role of SSL, we redesign an improved network architecture to Fully Exploit the limited capacity of neural Network (FE-Net). Equipped with SSL, this network can achieve 75.0% top-1 accuracy on ImageNet with only 563M M-Adds. It surpasses other counterparts constructed by depthwise separable convolution and the networks searched by NAS in terms of accuracy and practical speed.