# Rethinking Bottleneck Structure for Efficient Mobile Network Design

arXiv: https://arxiv.org/abs/2007.02269 · HTML: https://mlsanity.com/paper/2007.02269/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2020.

## Built on by (most cited first, 30 of 54)

- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Coordinate Attention for Efficient Mobile Network Design](https://mlsanity.com/paper/2103.02907/) (2021)
- [Mobile-Former: Bridging MobileNet and Transformer](https://mlsanity.com/paper/2108.05895/) (2021)
- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [Lite-HRNet: A Lightweight High-Resolution Network](https://mlsanity.com/paper/2104.06403/) (2021)
- [Constructing Stronger and Faster Baselines for Skeleton-based Action Recognition](https://mlsanity.com/paper/2106.15125/) (2021)
- [TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation](https://mlsanity.com/paper/2204.05525/) (2022)
- [Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition](https://mlsanity.com/paper/2211.11943/) (2022)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Rewrite the Stars](https://mlsanity.com/paper/2403.19967/) (2024)
- [GhostNetV2: Enhance Cheap Operation with Long-Range Attention](https://mlsanity.com/paper/2211.12905/) (2022)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [Revisiting Dynamic Convolution via Matrix Decomposition](https://mlsanity.com/paper/2103.08756/) (2021)
- [CondenseNet V2: Sparse Feature Reactivation for Deep Networks](https://mlsanity.com/paper/2104.04382/) (2021)
- [MicroNet: Improving Image Recognition with Extremely Low FLOPs](https://mlsanity.com/paper/2108.05894/) (2021)
- [GhostNets on Heterogeneous Devices via Cheap Operations](https://mlsanity.com/paper/2201.03297/) (2022)
- [A Guide to Image and Video based Small Object Detection using Deep Learning : Case Study of Maritime Surveillance](https://mlsanity.com/paper/2207.12926/) (2022)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [GhostNetV3: Exploring the Training Strategies for Compact Models](https://mlsanity.com/paper/2404.11202/) (2024)
- [RepGhost: A Hardware-Efficient Ghost Module via Re-parameterization](https://mlsanity.com/paper/2211.06088/) (2022)
- [MoCoViT: Mobile Convolutional Vision Transformer](https://mlsanity.com/paper/2205.12635/) (2022)
- [Dite-HRNet: Dynamic Lightweight High-Resolution Network for Human Pose Estimation](https://mlsanity.com/paper/2204.10762/) (2022)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets](https://mlsanity.com/paper/2310.10563/) (2023)
- [SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computers](https://mlsanity.com/paper/2311.03747/) (2023)
- [Lightweight Monocular Depth with a Novel Neural Architecture Search Method](https://mlsanity.com/paper/2108.11105/) (2021)
- [FalconNet: Factorization for the Light-weight ConvNets](https://mlsanity.com/paper/2306.06365/) (2023)
- [AutoSpace: Neural Architecture Search with Less Human Interference](https://mlsanity.com/paper/2103.11833/) (2021)
- [LCD-Net: A Lightweight Remote Sensing Change Detection Network Combining Feature Fusion and Gating Mechanism](https://mlsanity.com/paper/2410.11580/) (2024)

## Abstract (arXiv)

The inverted residual block is dominating architecture design for mobile networks recently. It changes the classic residual bottleneck by introducing two design rules: learning inverted residuals and using linear bottlenecks. In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion. We thus propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions and thus alleviates information loss and gradient confusion effectively. Extensive experiments demonstrate that, different from the common belief, such bottleneck structure is more beneficial than the inverted ones for mobile networks. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2. On Pascal VOC 2007 test set, we observe that there is also 0.9% mAP improvement in object detection. We further verify the effectiveness of the sandglass block by adding it into the search space of neural architecture search method DARTS. With 25% parameter reduction, the classification accuracy is improved by 0.13% over previous DARTS models. Code can be found at: https://github.com/zhoudaquan/rethinking_bottleneck_design.