mlsanityData to 7 Oct 2026

How Lightweight Can A Vision Transformer Be

arXiv 2407.17783 ↗First posted 25 Jul 2024cs.CV cs.AIcited by 1

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

In this paper, we explore a strategy that uses Mixture-of-Experts (MoE) to streamline, rather than augment, vision transformers. Each expert in an MoE layer is a SwiGLU feedforward network, where V and W2 are shared across the layer. No complex attention or convolutional mechanisms are employed. Depth-wise scaling is applied to progressively reduce the size of the hidden layer and the number of experts is increased in stages. Grouped query attention is used. We studied the proposed approach with and without pre-training on small datasets and investigated whether transfer learning works at this scale. We found that the architecture is competitive even at a size of 0.67M parameters.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 20 of its 28 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

20 of 28 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2016Layer Normalizationcited by 5,579
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
8 more
  1. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  2. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  3. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  4. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  5. 2022EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformerscited by 116
  6. 2020Bringing AI To Edge: From Deep Learning's Perspectivecited by 24
  7. 2023Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoderscited by 17
  8. 2024Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Imagescited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Lightweight Metadata-Aware Mixture-of-Experts Masked Autoencoder for Earth Observationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2022Residual Mixture of Experts
  2. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformers
  3. 2021Scaling Vision with Sparse Mixture of Experts
  4. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?
  5. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  6. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  7. 2023From Sparse to Soft Mixtures of Experts
  8. 2026CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection