mlsanityData to 14 Oct 2026

TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference

arXiv 2609.26061 ↗First posted 1 Aug 2026cs.NI cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-experts (MoE) models improve capacity with moderate overhead by sparsely activating experts per token. However, deploying MoE across resource-constrained edge servers incurs substantial cross-server communication as experts are distributed across heterogeneous servers. Existing placement methods optimize for raw token traffic, while conventional compression considers semantics but ignores topology-dependent routing costs. Consequently, independent optimization leads to inefficient communication and resource utilization. This paper proposes TopoCompress, a deployment- and topology-aware token compression framework for communication-efficient distributed edge MoE inference. It jointly optimizes token compression, expert deployment/replication, GPU-CPU residency, and collaborative routing to balance cross-server transmission, quality, and resource use. To address the coupling between token-level compression and epoch-level deployment, TopoCompress employs a two-timescale alternating optimization. In the online fast loop, it identifies and compresses low-importance, high-routing-cost tokens and jointly routes surviving expert activations. In the offline slow loop, it updates expert placement, replication, and GPU-CPU residency according to post-compression traffic accumulated during online inference. We establish the feasibility, optimality, convergence, and computational complexity. Simulations demonstrate that TopoCompress effectively reduces cross-server traffic and deployment resource consumption while maintaining controllable inference quality, enabling efficient distributed MoE inference over bandwidth- and resource-constrained edge infrastructures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 39 of its 45 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

39 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026OrderMoE: An expert similarity driven distributed edge MoE inference19 Jul
  2. 2025MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs22 Oct
  3. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement18 Aug
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  5. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  6. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  9. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  10. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 29

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2023A Survey of Large Language Modelscited by 2,576
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
17 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  2. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  4. 2023A Survey on Model Compression for Large Language Modelscited by 269
  5. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  6. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2019Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computingcited by 152
  10. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  11. 2019Convergence of Edge Computing and Deep Learning: A Comprehensive Surveycited by 120
  12. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  13. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 53
  14. 2021Optimal Model Placement and Online Model Splitting for Device-Edge Co-Inferencecited by 19
  15. 2024SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimizationcited by 11
  16. 2025Model-Distributed Inference for Large Language Models at the Edgecited by 8
  17. 2023Mobility and Cost Aware Inference Accelerating Algorithm for Edge Intelligencecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
  2. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference
  3. 2025Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference
  4. 2025Stable-MoE: Lyapunov-based Token Routing for Distributed Mixture-of-Experts Training over Edge Networks
  5. 2026Democratizing MoE inference on commodity GPUs with CoMoE
  6. 2026TAOT: Topology-Aware Optimal Transport for Dynamic Expert Replica Placement in MoE Training
  7. 2026Multi-Turn Distributed Inference with Mixture of Experts for 6G Edge--Cloud Networks
  8. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling