mlsanityData to 7 Oct 2026

WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

arXiv 2606.21868 ↗First posted 20 Jun 2026cs.LGcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Modern local and agentic workloads often need large-model capacity at low concurrency, but run on GPUs that cannot keep a frontier-scale model resident. Mixture-of-Experts (MoE) models are a natural fit because they activate only a small subset of experts per token, but their sparsity saves computation, not residency: the full expert pool still has to be stored, and any expert used by a layer must be in GPU memory when that layer runs. Static layer-level CPU offload makes such models fit, but transfers the expert layer in bulk on every forward pass, losing much of the sparsity advantage. We view low-resource MoE serving as a working-set problem on the GPU. Routed expert weights and the KV cache are two memory-demand streams competing for the same limited VRAM. We implement this view in WiSP (Working-Set Paging), a routing-aware expert pager that plugs into an unmodified serving engine and preserves byte-identical outputs. On a real 24 GiB RTX 3090, WiSP achieves up to 2.0x the decode throughput of static offload at the same memory budget when the model does not fit. A natural next step is to predict future experts and prefetch them. We find that this does not help in single-stream decode: the bottleneck is PCIe bandwidth, not prediction quality, so speculative transfers compete with demand transfers instead of hiding them. This shifts the design question from prefetching to allocation: how should one VRAM budget be divided between resident experts and the KV cache? We answer with MV-WSA (Marginal-Value Working-Set Allocation), which splits memory by marginal latency benefit per byte while enforcing a KV-admission floor. As a startup configurator, MV-WSA is the only policy we test that stays near-best on both prefill and decode; as a live controller, it resizes both pools while serving and reduces end-to-end time by up to 1.19x over a fixed offline split, without changing model outputs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 32 references that are in the map. First posted on arXiv 20 Jun 2026.

Builds on

24 of 32 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge19 Mar
  2. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  3. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  4. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  5. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  6. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  7. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 16

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  12. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
4 more
  1. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  2. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  3. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  4. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AnyJev Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  2. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
  3. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
  4. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  5. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  6. 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  8. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony