# Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks

arXiv: https://arxiv.org/abs/2607.02522 · HTML: https://mlsanity.com/paper/2607.02522/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 6 May 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 13 of 13)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [HexGen: Generative Inference of Large Language Model over Heterogeneous Environment](https://mlsanity.com/paper/2311.11514/) (2023)
- [WDMoE: Wireless Distributed Mixture of Experts for Large Language Models](https://mlsanity.com/paper/2411.06681/) (2024)
- [Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts](https://mlsanity.com/paper/2508.00234/) (2025)
- [Load is not what you should balance: Introducing Prequal](https://mlsanity.com/paper/2312.10172/) (2023)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures are increasingly deployed across 6G edge--cloud networks, where sparse activation reduces the computational footprint of each inference to only a fraction of the full expert set. However, MoE inference in edge-cloud networks creates a tension between KV state locality and elastic expert dispatch. KV state relocation incurs substantial transfer overhead, while expert computation benefits from spreading across the network to exploit available capacity. This tension is amplified in multi-turn inference, where each turn extends the KV state that must persist across the dialogue. To this end, we present StateFlow, a distributed inference policy that decouples persistent KV state from transient sparse computation. StateFlow pins KV state at a sticky serving site for cross-turn reuse and jointly optimizes expert dispatch and aggregation placement across the network. We further implement a real-world testbed with kernel-level network emulation and experimental results show that StateFlow sustains more than 2× higher stable dialogue concurrency than the distributed baseline solutions and reduces turn-level p95 latency under multi-turn inference by 53.0%.