mlsanityData to 7 Oct 2026

Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks

arXiv 2607.02522 ↗First posted 6 May 2026cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) architectures are increasingly deployed across 6G edge--cloud networks, where sparse activation reduces the computational footprint of each inference to only a fraction of the full expert set. However, MoE inference in edge-cloud networks creates a tension between KV state locality and elastic expert dispatch. KV state relocation incurs substantial transfer overhead, while expert computation benefits from spreading across the network to exploit available capacity. This tension is amplified in multi-turn inference, where each turn extends the KV state that must persist across the dialogue. To this end, we present StateFlow, a distributed inference policy that decouples persistent KV state from transient sparse computation. StateFlow pins KV state at a sticky serving site for cross-turn reuse and jointly optimizes expert dispatch and aggregation placement across the network. We further implement a real-world testbed with kernel-level network emulation and experimental results show that StateFlow sustains more than 2× higher stable dialogue concurrency than the distributed baseline solutions and reduces turn-level p95 latency under multi-turn inference by 53.0%.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 13 of its 18 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

13 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Models11 Nov

Other references · 12

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  7. 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 119
  8. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  9. 2023Stateful Large Language Model Serving with Pensievecited by 59
  10. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  11. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7
  12. 2023Load is not what you should balance: Introducing Prequalcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
  2. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inference
  3. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  4. 2026AceSpec: An Asymmetric Edge-Cloud Collaborative Framework for Communication-Efficient LLM Inference
  5. 2026EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
  6. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
  7. 2026OrderMoE: An expert similarity driven distributed edge MoE inference
  8. 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges