# MicroNet: Improving Image Recognition with Extremely Low FLOPs

arXiv: https://arxiv.org/abs/2108.05894 · HTML: https://mlsanity.com/paper/2108.05894/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 12 Aug 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Dynamic Convolution: Attention over Convolution Kernels](https://mlsanity.com/paper/1912.03458/) (2019)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [CondConv: Conditionally Parameterized Convolutions for Efficient Inference](https://mlsanity.com/paper/1904.04971/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [MixConv: Mixed Depthwise Convolutional Kernels](https://mlsanity.com/paper/1907.09595/) (2019)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)

## Built on by (most cited first, 26 of 26)

- [Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks](https://mlsanity.com/paper/2303.03667/) (2023)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [ParC-Net: Position Aware Circular Convolution with Merits from ConvNets and Transformer](https://mlsanity.com/paper/2203.03952/) (2022)
- [PaSTe: Improving the Efficiency of Visual Anomaly Detection at the Edge](https://mlsanity.com/paper/2410.11591/) (2024)
- [DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs](https://mlsanity.com/paper/2403.19588/) (2024)
- [Lightweight Monocular Depth with a Novel Neural Architecture Search Method](https://mlsanity.com/paper/2108.11105/) (2021)
- [FalconNet: Factorization for the Light-weight ConvNets](https://mlsanity.com/paper/2306.06365/) (2023)
- [DPNet: Dual-Path Network for Real-time Object Detection with Lightweight Attention](https://mlsanity.com/paper/2209.13933/) (2022)
- [eWaSR -- an embedded-compute-ready maritime obstacle detection network](https://mlsanity.com/paper/2304.11249/) (2023)
- [Self-Supervised Learning based on Heat Equation](https://mlsanity.com/paper/2211.13228/) (2022)
- [RepNeXt: A Fast Multi-Scale CNN using Structural Reparameterization](https://mlsanity.com/paper/2406.16004/) (2024)
- [An Ultra-low Power TinyML System for Real-time Visual Processing at Edge](https://mlsanity.com/paper/2207.04663/) (2022)
- [LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images](https://mlsanity.com/paper/2212.08490/) (2022)
- [Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance](https://mlsanity.com/paper/2305.16319/) (2023)
- [FedDAG: Federated Domain Adversarial Generation Towards Generalizable Medical Image Analysis](https://mlsanity.com/paper/2501.13967/) (2025)
- [Dynamic Shuffle: An Efficient Channel Mixture Method](https://mlsanity.com/paper/2310.02776/) (2023)
- [Exploring Invariance in Images through One-way Wave Equations](https://mlsanity.com/paper/2310.12976/) (2023)
- [A probabilistic framework for dynamic quantization](https://mlsanity.com/paper/2505.10689/) (2025)
- [CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation](https://mlsanity.com/paper/2511.19711/) (2025)
- [Fast Neural Architecture Search for Lightweight Dense Prediction Networks](https://mlsanity.com/paper/2203.01994/) (2022)
- [DRTAM: Dual Rank-1 Tensor Attention Module](https://mlsanity.com/paper/2203.05893/) (2022)
- [Fast-ParC: Capturing Position Aware Global Feature for ConvNets and ViTs](https://mlsanity.com/paper/2210.04020/) (2022)
- [OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning](https://mlsanity.com/paper/2401.11652/) (2024)
- [A lightweight YOLOv5-FFM model for occlusion pedestrian detection](https://mlsanity.com/paper/2408.06633/) (2024)
- [DeepTraverse: A Depth-First Search Inspired Network for Algorithmic Visual Understanding](https://mlsanity.com/paper/2506.10084/) (2025)
- [Rethinking Vision Transformer Depth via Structural Reparameterization](https://mlsanity.com/paper/2511.19718/) (2025)

## Abstract (arXiv)

This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e.g. 5M FLOPs on ImageNet classification). We found that two factors, sparse connectivity and dynamic activation function, are effective to improve the accuracy. The former avoids the significant reduction of network width, while the latter mitigates the detriment of reduction in network depth. Technically, we propose micro-factorized convolution, which factorizes a convolution matrix into low rank matrices, to integrate sparse connectivity into convolution. We also present a new dynamic activation function, named Dynamic Shift Max, to improve the non-linearity via maxing out multiple dynamic fusions between an input feature map and its circular channel shift. Building upon these two new operators, we arrive at a family of networks, named MicroNet, that achieves significant performance gains over the state of the art in the low FLOP regime. For instance, under the constraint of 12M FLOPs, MicroNet achieves 59.4% top-1 accuracy on ImageNet classification, outperforming MobileNetV3 by 9.6%. Source code is at hrefhttps://github.com/liyunsheng13/micronethttps://github.com/liyunsheng13/micronet.