mlsanityData to 7 Oct 2026

CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution

arXiv 2605.17889 ↗First posted 18 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture improves computational efficiency via sparse expert activation, but throughput-oriented inference faces substantial GPU memory pressure due to a significant parameter size and intermediate data. Prior works attempt to mitigate this using expert offloading with micro-batching or by offloading computation to the CPU. However, the fragmented workload resulting from micro-batching degrades operational intensity, causing expert execution to become memory-bound. Meanwhile, CPU offloading is constrained by slow PCIe transfers and its limited applicability to attention computation in the decode stage. Consequently, these inefficiencies prevent effective system utilization, severely restricting the end-to-end throughput of MoE inference. To address these challenges, this paper proposes CoX-MoE, an Advanced Matrix Extensions (AMX)-enabled CPU-GPU collaborative system that comprehensively optimizes MoE inference by combining coalesced expert execution with strategic workload orchestration for higher throughput. CoX-MoE introduces (i) a coalescing-aware orchestration policy to jointly optimize resource allocation by adopting ordinary batch, instead of micro-batch, for expert computation and selective attention offloading, and (ii) a static expert-aware stratification scheme that pre-assigns frequently activated experts to the GPU, mitigating PCIe transfer overhead and balancing workload for the CPU and GPU during inference. Compared to state-of-the-art frameworks, CoX-MoE delivers significant gains, achieving up to 7.1x and 2.4x higher throughput than FlexGen and MoE-Lightning, respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 10 of its 24 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

10 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models18 Dec
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  4. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 5

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  3. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
  4. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
  5. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  6. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  7. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
  8. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference