mlsanityData to 14 Oct 2026

NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding

arXiv 2605.21100 ↗First posted 20 May 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Modern serving systems for Mixture-of-Experts (MoE) models adopt hybrid data-expert parallelism: expert parallelism (EP) shards experts across GPUs to scale capacity, while data parallelism (DP) replicates attention layers across instances to process independent requests. Existing systems bind each request's attention, MoE communication, and KV cache to a single instance. Because attention latency scales with KV cache size while MoE communication latency scales with batch size, this binding cannot balance both simultaneously, producing EP stragglers; it also fragments KV memory across instances, inflating tail latency under long contexts. While existing context parallelism (CP) mitigates these constraints, its uniform parallelism degree incurs prohibitive communication and attention-side overheads. We present work, which decouples MoE communication from KV cache placement and achieves dual balance through dynamic context parallelism (DCP). DCP assigns each request a context-parallel degree sized to its KV footprint: long requests distribute attention across multiple instances; short requests remain local. This dynamic parallelism effectively liquefies the KV cache across the cluster, balancing both the per-instance KV cache occupancy and batch sizes without unnecessary load-balancing costs. To bridge DCP with static execution, work introduces an ahead-of-time (AOT) graph engine paired with a custom routing-based communication backend. Experimental results show that work maintains up to 1.88×--3.27× higher request rates under strict time-per-output-token (TPOT) service level objectives (SLOs). Furthermore, work significantly mitigates stragglers, reducing P99 tail latency by up to 1.79×--2.12×.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 41 of its 70 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

41 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL13 Mar
  2. 2025UCCL-EP: Portable Expert-Parallel Communication22 Dec
  3. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference15 Dec
  4. 2025fabric-lib: RDMA Point-to-Point Communication for LLM Systems31 Oct
  5. 2025MoEntwine: Unleashing the Potential of Wafer-scale Chips for Large-scale Expert Parallel Inference29 Oct
  6. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving22 Sep
  7. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding25 Jul
  8. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures14 May
  9. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts7 Mar
  10. 2025Semantic Parallelism: Redefining Efficient MoE Inference via Model-Data Co-Scheduling6 Mar
  11. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing10 Feb
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
1 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 28

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2024Mixtral of Expertscited by 1,823
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
16 more
  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  3. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
  4. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  5. 2025Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compilercited by 25
  6. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  7. 2025DeepServe: Serverless Large Language Model Serving at Scalecited by 18
  8. 2026State of AI: An Empirical 100 Trillion Token Study with OpenRoutercited by 18
  9. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  10. 2025MSCCL++: Rethinking GPU Communication Abstractions for AI Inferencecited by 10
  11. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  12. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingcited by 7
  13. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  14. 2025Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Trainingcited by 2
  15. 2025Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inferencecited by 2
  16. 2025Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelismcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
  2. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs
  3. 2026Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
  4. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processing
  5. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony
  6. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch
  7. 2026Surviving Partial Rank Failures in Wide Expert-Parallel MoE Inference
  8. 2026ExpertPlex: A High-Goodput Disaggregated Serving System for MoE LLMs with Adaptive Persistent Kernels