# NanoCP: Request-Level Dynamic Context Parallelism for Data-Expert Parallel Decoding

arXiv: https://arxiv.org/abs/2605.21100 · HTML: https://mlsanity.com/paper/2605.21100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 41 of its 70 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 30 of 41)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation](https://mlsanity.com/paper/2506.18095/) (2025)
- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Context Parallelism for Scalable Million-Token Inference](https://mlsanity.com/paper/2411.01783/) (2024)
- [MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing](https://mlsanity.com/paper/2502.06643/) (2025)
- [Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler](https://mlsanity.com/paper/2504.19442/) (2025)
- [FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism](https://mlsanity.com/paper/2412.01523/) (2024)
- [DeepServe: Serverless Large Language Model Serving at Scale](https://mlsanity.com/paper/2501.14417/) (2025)
- [State of AI: An Empirical 100 Trillion Token Study with OpenRouter](https://mlsanity.com/paper/2601.10088/) (2026)
- [Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts](https://mlsanity.com/paper/2503.05066/) (2025)
- [No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha](https://mlsanity.com/paper/2409.17264/) (2024)
- [Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving](https://mlsanity.com/paper/2509.17863/) (2025)
- [UCCL-EP: Portable Expert-Parallel Communication](https://mlsanity.com/paper/2512.19849/) (2025)
- [MSCCL++: Rethinking GPU Communication Abstractions for AI Inference](https://mlsanity.com/paper/2504.09014/) (2025)

## Abstract (arXiv)

Modern serving systems for Mixture-of-Experts (MoE) models adopt hybrid data-expert parallelism: expert parallelism (EP) shards experts across GPUs to scale capacity, while data parallelism (DP) replicates attention layers across instances to process independent requests. Existing systems bind each request's attention, MoE communication, and KV cache to a single instance. Because attention latency scales with KV cache size while MoE communication latency scales with batch size, this binding cannot balance both simultaneously, producing EP stragglers; it also fragments KV memory across instances, inflating tail latency under long contexts. While existing context parallelism (CP) mitigates these constraints, its uniform parallelism degree incurs prohibitive communication and attention-side overheads. We present work, which decouples MoE communication from KV cache placement and achieves dual balance through dynamic context parallelism (DCP). DCP assigns each request a context-parallel degree sized to its KV footprint: long requests distribute attention across multiple instances; short requests remain local. This dynamic parallelism effectively liquefies the KV cache across the cluster, balancing both the per-instance KV cache occupancy and batch sizes without unnecessary load-balancing costs. To bridge DCP with static execution, work introduces an ahead-of-time (AOT) graph engine paired with a custom routing-based communication backend. Experimental results show that work maintains up to 1.88×--3.27× higher request rates under strict time-per-output-token (TPOT) service level objectives (SLOs). Furthermore, work significantly mitigates stragglers, reducing P99 tail latency by up to 1.79×--2.12×.