mlsanityData to 13 Oct 2026

Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts

arXiv 2305.18691 ↗First posted 30 May 2023cs.AR cs.CVcited by 23

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Computer vision researchers are embracing two promising paradigms: Vision Transformers (ViTs) and Multi-task Learning (MTL), which both show great performance but are computation-intensive, given the quadratic complexity of self-attention in ViT and the need to activate an entire large MTL model for one task. M3ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE), where only a small portion of subnetworks ("experts") are sparsely and dynamically activated based on the current task. M3ViT achieves better accuracy and over 80% computation reduction but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations, including (1) a novel reordering mechanism for self-attention, which requires only constant bandwidth regardless of the target parallelism; (2) a fast single-pass softmax approximation; (3) an accurate and low-cost GELU approximation; (4) a unified and flexible computing unit that is shared by almost all computational layers to maximally reduce resource usage; and (5) uniquely for M3ViT, a novel patch reordering method to eliminate memory access overhead. Edge-MoE achieves 2.24x and 4.90x better energy efficiency comparing with GPU and CPU, respectively. A real-time video demonstration is available online, along with our open-source code written using High-Level Synthesis.

Key facts. Cited by 23 papers in the map (data through 13 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 26 of its 37 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

26 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design26 Oct

Other references · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  4. 2020End-to-End Object Detection with Transformerscited by 7,396
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,435
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,472
  8. 2021ViViT: A Video Vision Transformercited by 1,615
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  11. 2020Pre-Trained Image Processing Transformercited by 892
  12. 2018Taskonomy: Disentangling Task Transfer Learningcited by 633
13 more
  1. 2021Transformer in Transformercited by 633
  2. 2018End-to-End Multi-Task Learning with Attentioncited by 533
  3. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  4. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  5. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  6. 2019Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentationcited by 141
  7. 2018NDDR-CNN: Layerwise Feature Fusing in Multi-Task CNNs by Neural Discriminative Dimensionality Reductioncited by 127
  8. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  9. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
  10. 2020Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformercited by 43
  11. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 29
  12. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27
  13. 2021Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantificationcited by 21

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 9; 2026: 5.

23 papers.

Sort:

By area:

  1. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 68
  2. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  3. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  4. 2024A Survey of Small Language Modelscited by 31
  5. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 18
  6. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 18
  7. 2025Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learningcited by 14
  8. 2025TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAscited by 10
  9. 2024HLSFactory: A Framework Empowering High-Level Synthesis Datasets for Machine Learning and Beyondcited by 8
  10. 2024HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipelinecited by 7
  11. 2025UbiMoE: A Ubiquitous Mixture-of-Experts Vision Transformer Accelerator With Hybrid Computation Pattern on FPGAcited by 4
  12. 2025Understanding and Harnessing Sparsity in Unified Multimodal Modelscited by 4
11 more
  1. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  2. 2024When Foresight Pruning Meets Zeroth-Order Optimization: Efficient Federated Learning for Low-Memory Devicescited by 2
  3. 2025TeLLMe v2: An Efficient End-to-End Ternary LLM Prefill and Decode Accelerator with Table-Lookup Matmul on Edge FPGAscited by 2
  4. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  5. 2024InterroGate: Learning to Share, Specialize, and Prune Representations for Multi-task Learningcited by 1
  6. 2025CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGAcited by 0
  7. 2025Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devicescited by 0
  8. 2026ViM-Q: Scalable Algorithm-Hardware Co-Design for Vision Mamba Model Inference on FPGAcited by 0
  9. 2026ELiTeFormer: An Efficient Transformer for FPGAscited by 0
  10. 2026MoTE: Mixture of Task Experts for Multi-Task Video Understandingcited by 0
  11. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ButterflyViT: 354× Expert Compression for Edge Vision Transformers
  2. 2025Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning
  3. 2023From Sparse to Soft Mixtures of Experts
  4. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  5. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
  6. 2021Scaling Vision with Sparse Mixture of Experts
  7. 2024How Lightweight Can A Vision Transformer Be
  8. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning