mlsanityData to 7 Oct 2026

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

arXiv 2604.02715 ↗First posted 3 Apr 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new expert paging abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8×H20 GPUs, FluxMoE delivers up to 7.2× vLLM's throughput and 79.0% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8×7B-Instruct on 2×L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3× KTransformers's throughput and 29.1% lower average TPOT.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 29 of its 44 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

29 of 44 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models30 Jan
  2. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling29 Jan
  3. 2025eMoE: Task-aware Memory Efficient Mixture-of-Experts-Based (MoE) Model Inference10 Mar
  4. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  5. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  7. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  9. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 20

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  10. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  12. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
8 more
  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  2. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  3. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  4. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  5. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  6. 2024ZipNN: Lossless Compression for AI Modelscited by 12
  7. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  8. 2025Lossless Compression of Neural Network Components: Weights, Checkpoints, and K/V Caches in Low-Precision Formatscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
  2. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  3. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  4. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
  5. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  6. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  8. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism