mlsanityData to 7 Oct 2026

Navigating Efficiency in MobileViT through Gaussian Process on Global Architecture Factors

arXiv 2406.04820 ↗First posted 7 Jun 2024cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Numerous techniques have been meticulously designed to achieve optimal architectures for convolutional neural networks (CNNs), yet a comparable focus on vision transformers (ViTs) has been somewhat lacking. Despite the remarkable success of ViTs in various vision tasks, their heavyweight nature presents challenges of computational costs. In this paper, we leverage the Gaussian process to systematically explore the nonlinear and uncertain relationship between performance and global architecture factors of MobileViT, such as resolution, width, and depth including the depth of in-verted residual blocks and the depth of ViT blocks, and joint factors including resolution-depth and resolution-width. We present design principles twisting magic 4D cube of the global architecture factors that minimize model sizes and computational costs with higher model accuracy. We introduce a formula for downsizing architectures by iteratively deriving smaller MobileViT V2, all while adhering to a specified constraint of multiply-accumulate operations (MACs). Experiment results show that our formula significantly outperforms CNNs and mobile ViTs across diversified datasets

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

42 of 51 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2019Searching for MobileNetV3cited by 1,984
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
30 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  9. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  11. 2021Early Convolutions Help Transformers See Bettercited by 395
  12. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  13. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  14. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  15. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  16. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  17. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  18. 2017Hierarchical Representations for Efficient Architecture Searchcited by 188
  19. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  20. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  21. 2019Towards Optimal Structured CNN Pruning via Generative Adversarial Learningcited by 145
  22. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  23. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  24. 2020Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNetscited by 49
  25. 2019Customizable Architecture Search for Semantic Segmentationcited by 41
  26. 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
  27. 2018Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagationcited by 29
  28. 2019Positive-Unlabeled Compression on the Cloudcited by 27
  29. 2020Gaussian Processes with Skewed Laplace Spectral Mixture Kernels for Long-term Forecastingcited by 3
  30. 2018Compressible Spectral Mixture Kernels with Sparse Dependency Structures for Gaussian Processescited by 2

Built on by

Not cited yet.