mlsanityData to 14 Oct 2026

Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling

arXiv 2603.27624 ↗First posted 29 Mar 2026cs.AR cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts is a promising approach for edge AI with low-batch inference. Yet, on-device deployments often face limited on-chip memory and severe workload imbalance; the prevalent use of offloading further incurs off-chip memory access bottlenecks. Moreover, MoE sparsity and dynamic gating shift distributed strategies toward much finer granularity and introduce runtime scheduling considerations. Recently, high die-to-die bandwidth chiplet interconnects have created new opportunities for multi-chiplet systems to address workload imbalance and offloading bottlenecks with fine-grained scheduling. In this paper, we propose Fully Sharded Expert Data Parallelism, a parallelization paradigm specifically architected for low-batch MoE inference on multi-chiplet accelerators. FSE-DP attains adaptive computation-communication overlap and balanced load by orchestrating fine-grained, complementary expert streams along dynamic trajectories across high-bandwidth D2D links. The attendant dataflow complexity is tamed by a minimal, hardware-amenable set of virtualization rules and a lightweight scheduling algorithm. Our approach achieves 1.22 to 2.00 times speedup over state-of-the-art baselines and saves up to 78.8 percent on-chip memory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 57 of its 89 references that are in the map. First posted on arXiv 29 Mar 2026.

Builds on

57 of 89 references.

Closest ancestors · 29

Same direction, newest first

  1. 2025HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission22 Oct
  2. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference7 Oct
  3. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving22 Sep
  4. 2025HarMoEny: Efficient Multi-GPU Inference of MoE Models14 Jun
  5. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving17 Apr
  6. 2025MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism3 Apr
  7. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts7 Mar
  8. 2025Muon is Scalable for LLM Training24 Feb
  9. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  10. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  11. 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference27 Jan
  12. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models18 Jan
17 more
  1. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  3. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  5. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  6. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  7. 2024MoNDE: Mixture of Near-Data Experts for Large-Scale Sparse Models29 May
  8. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  11. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  12. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism22 Apr
  13. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training11 Mar
  14. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  15. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  16. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  17. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 28

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  8. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  10. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  11. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  12. 2025DeepSeek-OCR: Contexts Optical Compressioncited by 183
16 more
  1. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  3. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  4. 2024MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generationcited by 51
  5. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
  6. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  7. 2024GRIN: GRadient-INformed MoEcited by 17
  8. 2023Optimizing Distributed ML Communication with Fused Computation-Collective Operationscited by 15
  9. 2025WaferLLM: Large Language Model Inference at Wafer Scalecited by 13
  10. 20253D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generationcited by 9
  11. 2024Scaling Deep Learning Computation over the Inter-Core Connected Intelligence Processor with T10cited by 5
  12. 2025The MoE-Empowered Edge LLMs Deployment: Architecture, Challenges, and Opportunitiescited by 5
  13. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  14. 2022Optimizing Mixture of Experts using Dynamic Recompilationscited by 4
  15. 2025A 28nm 0.22μJ/token memory-compute-intensity-aware CNN-Transformer accelerator with hybrid-attention-based layer-fusion and cascaded pruning for semantic-segmentationcited by 3
  16. 2025MLDSE: Scaling Design Space Exploration Infrastructure for Multi-Level Hardwarecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony
  2. 2026HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference
  3. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
  6. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
  7. 2026ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts
  8. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios