# Rethinking Mobile Block for Efficient Attention-based Models

arXiv: https://arxiv.org/abs/2301.01146 · HTML: https://mlsanity.com/paper/2301.01146/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 89 references that are in the map.
- First posted on arXiv 3 Jan 2023.

## Builds on (most cited first, 30 of 76)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)

## Built on by (most cited first, 30 of 36)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training](https://mlsanity.com/paper/2411.16061/) (2024)
- [SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design](https://mlsanity.com/paper/2401.16456/) (2024)
- [Exploring Plain ViT Reconstruction for Multi-class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2312.07495/) (2023)
- [EdgeSAM: Prompt-In-the-Loop Distillation for SAM](https://mlsanity.com/paper/2312.06660/) (2023)
- [DGMamba: Domain Generalization via Generalized State Space Model](https://mlsanity.com/paper/2404.07794/) (2024)
- [Mamba or RWKV: Exploring High-Quality and High-Efficiency Segment Anything Model](https://mlsanity.com/paper/2406.19369/) (2024)
- [EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality](https://mlsanity.com/paper/2411.15241/) (2024)
- [Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark](https://mlsanity.com/paper/2404.10760/) (2024)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)
- [A Generalist FaceX via Learning Unified Facial Representation](https://mlsanity.com/paper/2401.00551/) (2023)
- [CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications](https://mlsanity.com/paper/2408.03703/) (2024)
- [MobileIE: An Extremely Lightweight and Effective ConvNet for Real-Time Image Enhancement on Mobile Devices](https://mlsanity.com/paper/2507.01838/) (2025)
- [SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computers](https://mlsanity.com/paper/2311.03747/) (2023)
- [iFormer: Integrating ConvNet and Transformer for Mobile Application](https://mlsanity.com/paper/2501.15369/) (2025)
- [A Comprehensive Library for Benchmarking Multi-class Visual Anomaly Detection](https://mlsanity.com/paper/2406.03262/) (2024)
- [CCi-YOLOv8n: Enhanced Fire Detection with CARAFE and Context-Guided Modules](https://mlsanity.com/paper/2411.11011/) (2024)
- [RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything](https://mlsanity.com/paper/2401.10228/) (2024)
- [Fully Spiking Neural Networks for Unified Frame-Event Object Tracking](https://mlsanity.com/paper/2505.20834/) (2025)
- [CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction](https://mlsanity.com/paper/2411.16170/) (2024)
- [RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers](https://mlsanity.com/paper/2505.21847/) (2025)
- [MobileUtr: Revisiting the relationship between light-weight CNN and Transformer for efficient medical image segmentation](https://mlsanity.com/paper/2312.01740/) (2023)
- [SRSNetwork: Siamese Reconstruction-Segmentation Networks based on Dynamic-Parameter Convolution](https://mlsanity.com/paper/2312.01741/) (2023)
- [VQualA 2025 Challenge on Face Image Quality Assessment: Methods and Results](https://mlsanity.com/paper/2508.18445/) (2025)
- [Exploring Real&Synthetic Dataset and Linear Attention in Image Restoration](https://mlsanity.com/paper/2412.03814/) (2024)
- [S2AFormer: Strip Self-Attention for Efficient Vision Transformer](https://mlsanity.com/paper/2505.22195/) (2025)
- [UniConvNet: Expanding Effective Receptive Field while Maintaining Asymptotically Gaussian Distribution for ConvNets of Any Scale](https://mlsanity.com/paper/2508.09000/) (2025)
- [VividAnimator: An End-to-End Audio and Pose-driven Half-Body Human Animation Framework](https://mlsanity.com/paper/2510.10269/) (2025)

## Abstract (arXiv)

This paper focuses on developing modern, efficient, lightweight models for dense predictions while trading off parameters, FLOPs, and performance. Inverted Residual Block (IRB) serves as the infrastructure for lightweight CNNs, but no counterpart has been recognized by attention-based studies. This work rethinks lightweight infrastructure from efficient IRB and effective components of Transformer from a unified perspective, extending CNN-based IRB to attention-based models and abstracting a one-residual Meta Mobile Block (MMB) for lightweight model design. Following simple but effective design criterion, we deduce a modern Inverted Residual Mobile Block (iRMB) and build a ResNet-like Efficient MOdel (EMO) with only iRMB for down-stream tasks. Extensive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, e.g., EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass equal-order CNN-/Attention-based models, while trading-off the parameter, efficiency, and accuracy well: running 2.8-4.0x faster than EdgeNeXt on iPhone14.