# Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks

arXiv: https://arxiv.org/abs/2303.03667 · HTML: https://mlsanity.com/paper/2303.03667/

## Key facts

- Cited by 130 papers in the map (data through 14 Oct 2026).
- Builds on 75 of its 89 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 30 of 75)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)

## Built on by (most cited first, 30 of 130)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design](https://mlsanity.com/paper/2401.16456/) (2024)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [The Tenth NTIRE 2025 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2504.10686/) (2025)
- [Learned Image Compression with Hierarchical Progressive Context Modeling](https://mlsanity.com/paper/2507.19125/) (2025)
- [Efficient Modulation for Vision Networks](https://mlsanity.com/paper/2403.19963/) (2024)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)
- [LSNet: See Large, Focus Small](https://mlsanity.com/paper/2503.23135/) (2025)
- [Fostc3net:A Lightweight YOLOv5 Based On the Network Structure Optimization](https://mlsanity.com/paper/2403.13703/) (2024)
- [MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices](https://mlsanity.com/paper/2507.01838/) (2025)
- [SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention](https://mlsanity.com/paper/2407.05128/) (2024)
- [LWGANet: Addressing Spatial and Channel Redundancy in Remote Sensing Visual Tasks with Light-Weight Grouped Attention](https://mlsanity.com/paper/2501.10040/) (2025)
- [Real-World Remote Sensing Image Dehazing: Benchmark and Baseline](https://mlsanity.com/paper/2503.17966/) (2025)
- [Rapid Salient Object Detection with Difference Convolutional Neural Networks](https://mlsanity.com/paper/2507.01182/) (2025)
- [RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets](https://mlsanity.com/paper/2310.10563/) (2023)
- [Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection](https://mlsanity.com/paper/2405.16038/) (2024)
- [Pragmatic Heterogeneous Collaborative Perception via Generative Communication Mechanism](https://mlsanity.com/paper/2510.19618/) (2025)
- [MicroViT: A Vision Transformer with Low Complexity Self Attention for Edge Device](https://mlsanity.com/paper/2502.05800/) (2025)
- [FGENet: Fine-Grained Extraction Network for Congested Crowd Counting](https://mlsanity.com/paper/2401.01208/) (2024)
- [Partial Large Kernel CNNs for Efficient Super-Resolution](https://mlsanity.com/paper/2404.11848/) (2024)
- [FalconNet: Factorization for the Light-weight ConvNets](https://mlsanity.com/paper/2306.06365/) (2023)
- [LDConv: Linear deformable convolution for improving convolutional neural networks](https://mlsanity.com/paper/2311.11587/) (2023)
- [LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones](https://mlsanity.com/paper/2409.03460/) (2024)
- [MOFA: A Model Simplification Roadmap for Image Restoration on Mobile Devices](https://mlsanity.com/paper/2308.12494/) (2023)
- [PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer](https://mlsanity.com/paper/2407.11306/) (2024)
- [AI Governance through Markets](https://mlsanity.com/paper/2501.17755/) (2025)
- [UltraFast-LiNET: Light-weight multi-scale shift convolutional network for real-time low-light image enhancement](https://mlsanity.com/paper/2512.02965/) (2025)
- [Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensing](https://mlsanity.com/paper/2309.06047/) (2023)
- [ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effect](https://mlsanity.com/paper/2401.12736/) (2024)

## Abstract (arXiv)

To design fast neural networks, many works have been focusing on reducing the number of floating-point operations (FLOPs). We observe that such reduction in FLOPs, however, does not necessarily lead to a similar level of reduction in latency. This mainly stems from inefficiently low floating-point operations per second (FLOPS). To achieve faster networks, we revisit popular operators and demonstrate that such low FLOPS is mainly due to frequent memory access of the operators, especially the depthwise convolution. We hence propose a novel partial convolution (PConv) that extracts spatial features more efficiently, by cutting down redundant computation and memory access simultaneously. Building upon our PConv, we further propose FasterNet, a new family of neural networks, which attains substantially higher running speed than others on a wide range of devices, without compromising on accuracy for various vision tasks. For example, on ImageNet-1k, our tiny FasterNet-T0 is 2.8×, 3.3×, and 2.4× faster than MobileViT-XXS on GPU, CPU, and ARM processors, respectively, while being 2.9% more accurate. Our large FasterNet-L achieves impressive 83.5% top-1 accuracy, on par with the emerging Swin-B, while having 36% higher inference throughput on GPU, as well as saving 37% compute time on CPU. Code is available at urlhttps://github.com/JierunChen/FasterNet.