mlsanityData to 14 Oct 2026

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

arXiv 2606.10493 ↗First posted 9 Jun 2026cs.DC cs.AI cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads. We identify four key gaps in local MoE inference: reliance on capacity-reduced models (quantized, distilled, rerouted), inability to meet 30-second TTFT for long prefills (more than 12K), sub-baseline decode throughput (under 20 tokens/s), and poor concurrency under mixed prefill-decode and batched decode workloads. We present a CPU-GPU hybrid system that achieves cloud-level SLOs on dual-socket commodity CPUs and consumer GPUs by (1) stream-loading prefill (SLP), boosting prefill throughput to 1,200 tokens/s and enabling 32K prompts within 30 seconds; (2) distributed SLP (DSLP) with SmallEP expert parallelism, reaching 1,800 tokens/s and 45K prompts in 30 seconds on two RTX 5090s; (3) intra-node prefill-decode disaggregation with zero-copy shared weights and a dual-batch attention-MoE overlap scheme, sustaining concurrency with under 15 percent latency increase and 50 percent throughput gains; (4) an AVX-512-optimized FP8 GEMV kernel, enabling native CPU FP8 inference while delivering 4-5x lower CPU latency; and (5) fine-grained CPU parallelism that attains 28 tokens/s on INT4 DeepSeek-V3 and 21.5 tokens/s on intact FP8 V3. Evaluations show our system delivers cloud-level QoS for flagship MoE models on consumer CPU-GPU platforms, reshaping local deployment with intact, original-precision inference and enabling high-quality, cost-effective access without datacenter infrastructure.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 25 of its 59 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

25 of 59 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul
  2. 2025Serving Large Language Models on Huawei CloudMatrix38415 Jun
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  4. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  5. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling23 Oct
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb

Other references · 19

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2024DeepSeek-V3 Technical Reportcited by 811
  8. 2025Kimi K2: Open Agentic Intelligencecited by 567
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  11. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  12. 2022FP8 Formats for Deep Learningcited by 196
7 more
  1. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  3. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  4. 2024Scaling FP8 training to trillion-token LLMscited by 65
  5. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  6. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  7. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0
  2. 2026Cascadia: Resident 975B MoE Inference on Eleven AI PCscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference
  2. 2024Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling
  3. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  4. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  5. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  6. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
  7. 2026Communication-Aware Placement and Pruning for Efficient Mixture-of-Experts Inference
  8. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading