Exploring and Improving Mobile Level Vision Transformers
Not on the map. Why?
Abstract (arXiv)
We study the vision transformer structure in the mobile level in this paper, and find a dramatic performance drop. We analyze the reason behind this phenomenon, and propose a novel irregular patch embedding module and adaptive patch fusion module to improve the performance. We conjecture that the vision transformer blocks (which consist of multi-head attention and feed-forward network) are more suitable to handle high-level information than low-level features. The irregular patch embedding module extracts patches that contain rich high-level information with different receptive fields. The transformer blocks can obtain the most useful information from these irregular patches. Then the processed patches pass the adaptive patch merging module to get the final features for the classifier. With our proposed improvements, the traditional uniform vision transformer structure can achieve state-of-the-art results in mobile level. We improve the DeiT baseline by more than 9% under the mobile-level settings and surpass other transformer architectures like Swin and CoaT by a large margin.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 44 of its 53 references that are in the map. First posted on arXiv 30 Aug 2021.
Builds on
44 of 53 references.
References · 44
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
32 more
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2018CBAM: Convolutional Block Attention Modulecited by 2,477
- 2017Non-local Neural Networkscited by 2,293
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2019Searching for MobileNetV3cited by 1,994
- 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2017Random Erasing Data Augmentationcited by 1,239
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
- 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
- 2016Deep Networks with Stochastic Depthcited by 979
- 2020Designing Network Design Spacescited by 960
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
- 2018Image Transformercited by 772
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
- 2021Transformer in Transformercited by 633
- 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
- 2020ResNeSt: Split-Attention Networkscited by 445
- 2019Selective Kernel Networkscited by 403
- 2021Early Convolutions Help Transformers See Bettercited by 396
- 2017Residual Attention Network for Image Classificationcited by 386
- 2020Are we done with ImageNet?cited by 333
- 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
- 2021Co-Scale Conv-Attentional Image Transformerscited by 224
- 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
- 2021ConTNet: Why not use convolution and transformer at the same time?cited by 38
Built on by
Citing papers per year: 2021: 2.
2 papers.