mlsanityData to 7 Oct 2026

A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Accelerators

arXiv 2609.21137 ↗First posted 17 Sep 2026cs.ARcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) decoding on scratchpad-based tensor accelerators (STA) is dominated by moving expert weights while the compute engines sit idle. This traffic is hard to hide, because the experts are known only after routing, and hard to shrink without losing quality or adding critical-path work. We present CARDAN, which represents each expert-weight matrix as a vector-quantized component plus a shared-basis low-rank component and co-designs this representation with a multi-engine decoding dataflow. The representation separates expert-common from expert-private work, so the dataflow overlaps DMA with computation on several engines. Across five MoE families on AWS Trainium3, CARDAN matches or improves BF16-teacher perplexity across all five models and speeds up batch-one decoding by 1.15-1.31x over AWS dense MoE megakernels, rising to 1.7x at batch size 16.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 51 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

27 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  2. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2 Sep
  3. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
  4. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  6. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 20

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  4. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  7. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  8. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  9. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  10. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  11. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  12. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
8 more
  1. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 48
  2. 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 39
  3. 2024QERA: an Analytical Framework for Quantization Error Reconstructioncited by 21
  4. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  5. 2025MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Modelscited by 7
  6. 2021Extending Sparse Tensor Accelerators to Support Multiple Compression Formatscited by 5
  7. 2023A High-performance, Energy-efficient Modular DMA Engine Architecturecited by 4
  8. 2026RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compressioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding
  2. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models
  3. 2025Accelerating MoE Model Inference with Expert Sharding
  4. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
  5. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
  6. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
  7. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
  8. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training