mlsanityData to 14 Oct 2026

CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGA

arXiv 2506.08496 ↗First posted 10 Jun 2025cs.ARcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Vision Transformers (ViTs) exhibit superior performance in computer vision tasks but face deployment challenges on resource-constrained devices due to high computational/memory demands. While Mixture-of-Experts Vision Transformers (MoE-ViTs) mitigate this through a scalable architecture with sub-linear computational growth, their hardware implementation on FPGAs remains constrained by resource limitations. This paper proposes a novel accelerator for efficiently implementing quantized MoE models on FPGAs through two key innovations: (1) A dual-stage quantization scheme combining precision-preserving complex quantizers with hardware-friendly simplified quantizers via scale reparameterization, with only 0.28 % accuracy loss compared to full precision; (2) A resource-aware accelerator architecture featuring latency-optimized streaming attention kernels and reusable linear operators, effectively balancing performance and resource consumption. Experimental results demonstrate that our accelerator achieves nearly 155 frames per second, a 5.35× improvement in throughput, and over 80% energy reduction compared to state-of-the-art (SOTA) FPGA MoE accelerators, while maintaining <1% accuracy loss across vision benchmarks. Our implementation is available at https://github.com/DJ000011/CoQMoE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 19 of its 26 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

19 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025UbiMoE: A Ubiquitous Mixture-of-Experts Vision Transformer Accelerator With Hybrid Computation Pattern on FPGA8 Feb
  2. 2023Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts30 May
  3. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design26 Oct
  4. 2021Go Wider Instead of Deeper25 Jul
  5. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 14

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2021Post-Training Quantization for Vision Transformercited by 278
  6. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  7. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 137
  8. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  9. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 56
  10. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  11. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Modelscited by 11
  12. 2023TSPTQ-ViT: Two-scaled post-training quantization for vision transformercited by 7
2 more
  1. 2024HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipelinecited by 7
  2. 2024P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformercited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer
  2. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  3. 2024M2-ViT: Accelerating Hybrid Vision Transformers with Two-Level Mixed Quantization
  4. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
  5. 2024Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
  6. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  7. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operations
  8. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer