mlsanityData to 14 Oct 2026

ButterflyViT: 354× Expert Compression for Edge Vision Transformers

arXiv 2603.06746 ↗First posted 6 Mar 2026cs.CV cs.AIcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Deploying sparse Mixture of Experts(MoE) Vision Transformers remains a challenge due to linear expert memory scaling. Linear memory scaling stores N independent expert weight matrices requiring O(NE · d2) memory, which exceeds edge devices memory budget. Current compression methods like quantization, pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyViT, a method that treats experts not as independent weight matrices but as geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields O(dmodel · dff + NE · nℓ · d) memory which is sub-linear in the number of experts. To address the unique challenges of vision, a spatial smoothness regulariser is introduced that penalises routing irregularities between adjacent patch tokens, turning patch correlation into a training signal. Across image classification tasks on CIFAR-100, ButterflyViT achieves 354× memory reduction at 64 experts with negligible accuracy loss. ButterflyViT allows multiple experts to fit on edge-constrained devices showing that geometric parameterization breaks linear scaling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 24 of its 27 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

24 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts8 Sep
  2. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design26 Oct
  3. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  9. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 471
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2021Post-Training Quantization for Vision Transformercited by 278
9 more
  1. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  2. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 179
  3. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  4. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  5. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  6. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  7. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  8. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  9. 2022O-ViT: Orthogonal Vision Transformercited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits
  2. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformers
  3. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  4. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts
  5. 2025EdgeFlex-Transformer: Transformer Inference for Edge Devices
  6. 2023From Sparse to Soft Mixtures of Experts
  7. 2026Scaling Vision Transformers Without Training: Prune-and-Merge Token Compression
  8. 2023CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs