mlsanityData to 7 Oct 2026

HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing

arXiv 2609.08682 ↗First posted 8 Sep 2026cs.ARcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures have become a key technique for scaling Large Language Models (LLMs), enabling high model capacity with reduced computational cost. However, this efficiency comes at the expense of increased memory capacity and bandwidth demands. Recent 3D Near-Memory Processing (NMP) architectures, which vertically integrate memory and compute through hybrid bonding, provide high internal bandwidth and energy efficiency, making them attractive for accelerating MoE inference. Nevertheless, the distributed memory and compute organization of NMP systems introduces new challenges for mapping MoE workloads. Existing parallelization strategies, such as Tensor Parallelism (TP) and Expert Parallelism (EP), suffer from either high communication costs or unbalanced computation utilization, leading to inferior efficiency. In addition, the dynamic routing behavior of MoE models further complicates efficient deployment. To address these challenges, we present HDA-MoE, a framework that optimizes MoE execution on NMP architectures through hybrid parallel deployment and runtime scheduling. HDA-MoE integrates an offline hybrid parallel mapping algorithm with an online dynamic and adaptive scheduling mechanism to reduce communication overhead while improving computation utilization. Experimental results show that HDA-MoE achieves a speedup of 1.1x--3.4x over TP, 1.1x--1.5x over EP, 1.1x--3.7x over the Hybrid TP-EP compute-balanced baseline, and 1.1x--1.3x over HD-MoE. Source code is available at https://github.com/PKU-SEC-Lab/HDA-MoE-TCAD26.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 49 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

27 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  2. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference13 Nov
  3. 2025HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap13 Aug
  4. 2025A3D-MoE: Acceleration of Large Language Models with Mixture of Experts via 3D Heterogeneous Integration25 Jul
  5. 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
  6. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  7. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  8. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  9. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  10. 2024EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference16 Oct
  11. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  12. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts7 Apr

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2024Mixtral of Expertscited by 1,785
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
3 more
  1. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  2. 2021TACCL: Guiding Collective Algorithm Synthesis using Communication Sketchescited by 38
  3. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Modelscited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  2. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  3. 2026DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
  4. 2023Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism
  5. 2026MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm
  6. 2025HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference
  7. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  8. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems