mlsanityData to 14 Oct 2026

SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

arXiv 2609.34117 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-experts (MoE) models activate few experts per token, yet batched decoding can access nearly the entire expert pool, making expert-weight traffic a major bottleneck. Expert pruning reduces this traffic, but conventional approaches also prune compute-bound prefill, sacrificing model quality for little throughput benefit. We present SlimWise, a serving framework that tailors the expert pool to each inference phase. SlimWise performs prefill with the full model and decode with a pruned model that directly reuses the prefill-generated KV cache without conversion. Across two MoE backbones and three pruning criteria, this training-free KV cache handoff substantially narrows accuracy gaps relative to the full model in many settings. We also show that benchmark accuracy can conceal substantial pruning-induced changes in generation length. To address these distortions and residual accuracy loss, SlimWise introduces a low-cost distillation stage that trains the decoder to continue from full-model KV caches while updating only a small subset of parameters. Implemented in vLLM, SlimWise supports both prefill-decode (PD) disaggregation and PD-colocated serving. On Qwen3.6-35B-A3B, SlimWise improves decode throughput by up to 1.81x at 50% expert pruning with minimal accuracy loss.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 36 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

29 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration24 Aug
  2. 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving1 Jul
  3. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference6 Feb
  4. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inference15 Dec
  5. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov

Other references · 24

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2024Mixtral of Expertscited by 1,823
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  12. 2026Gemma 4 Technical Reportcited by 259
12 more
  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  2. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  4. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  5. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  6. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  7. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  8. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  9. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Notcited by 10
  10. 2025Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retrainingcited by 7
  11. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  12. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  2. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  3. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
  4. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  5. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
  6. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  7. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  8. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference