# Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions

arXiv: https://arxiv.org/abs/1711.08141 · HTML: https://mlsanity.com/paper/1711.08141/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2017.

## Built on by (most cited first, 30 of 93)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Efficient Long-Range Attention Network for Image Super-resolution](https://mlsanity.com/paper/2203.06697/) (2022)
- [AS-MLP: An Axial Shifted MLP Architecture for Vision](https://mlsanity.com/paper/2107.08391/) (2021)
- [Space-time Mixing Attention for Video Transformer](https://mlsanity.com/paper/2106.05968/) (2021)
- [S2-MLP: Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2106.07477/) (2021)
- [A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift](https://mlsanity.com/paper/2206.10810/) (2022)
- [Token Shift Transformer for Video Classification](https://mlsanity.com/paper/2108.02432/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions](https://mlsanity.com/paper/2111.08446/) (2021)
- [FEAR: Fast, Efficient, Accurate and Robust Visual Tracker](https://mlsanity.com/paper/2112.07957/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [MixFaceNets: Extremely Efficient Face Recognition Networks](https://mlsanity.com/paper/2107.13046/) (2021)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [EdgeFace: Efficient Face Recognition Model for Edge Devices](https://mlsanity.com/paper/2307.01838/) (2023)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism](https://mlsanity.com/paper/2201.10801/) (2022)
- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning](https://mlsanity.com/paper/2110.10538/) (2021)
- [Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion](https://mlsanity.com/paper/2108.05009/) (2021)
- [DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks](https://mlsanity.com/paper/2206.00843/) (2022)
- [DyFADet: Dynamic Feature Aggregation for Temporal Action Detection](https://mlsanity.com/paper/2407.03197/) (2024)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives](https://mlsanity.com/paper/2308.02066/) (2023)
- [TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device](https://mlsanity.com/paper/2109.13227/) (2021)
- [HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning](https://mlsanity.com/paper/2106.08630/) (2021)
- [Rethinking Token-Mixing MLP for MLP-based Vision Backbone](https://mlsanity.com/paper/2106.14882/) (2021)
- [GhostNets on Heterogeneous Devices via Cheap Operations](https://mlsanity.com/paper/2201.03297/) (2022)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)

## Abstract (arXiv)

Neural networks rely on convolutions to aggregate spatial information. However, spatial convolutions are expensive in terms of model size and computation, both of which grow quadratically with respect to kernel size. In this paper, we present a parameter-free, FLOP-free "shift" operation as an alternative to spatial convolutions. We fuse shifts and point-wise convolutions to construct end-to-end trainable shift-based modules, with a hyperparameter characterizing the tradeoff between accuracy and efficiency. To demonstrate the operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters; we additionally demonstrate the operation's resilience to parameter reduction on ImageNet, outperforming ResNet family members. We finally show the shift operation's applicability across domains, achieving strong performance with fewer parameters on classification, face verification and style transfer.