mlsanityData to 14 Oct 2026

Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

arXiv 2608.15383 ↗First posted 15 Aug 2026cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank. We present ExactMoE, an inference design that applies symmetric group-128 four-bit weight quantization only to routed experts, stores those experts in kernel-native MARLIN form in pinned host memory, and executes all selected experts through a configurable GPU-resident slot cache and fused grouped MoE kernels. The router, attention, embeddings, normalization layers, and language-model head remain in BF16. "Exact" refers to complete expert availability and an unchanged top-k routing procedure: no expert is pruned, substituted, or forced to execute on the CPU. It does not imply numerical identity with the BF16 model. On OLMoE-1B-7B-0924-Instruct, evaluated on a single NVIDIA L4, a 16-slot configuration reduces peak reserved GPU memory from 14.168 to 1.836 GiB (87.04%) while retaining 81.85% of BF16 decode throughput. A fully resident 64-slot configuration reaches 31.923 tokens/s versus 21.662 tokens/s for BF16 while reserving 4.061 GiB. Across 12,450 zero-shot multiple-choice questions, ExactMoE obtains 70.3534% normalized accuracy versus 70.8996% for BF16, retaining 99.23% of the baseline accuracy. In a matched 16-token ablation, fused grouped execution is 1.97x as fast as a sequential W4 reference. These results identify a practical memory-transfer-throughput frontier for complete-expert MoE inference.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 25 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

21 of 25 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization17 Jul
  2. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference19 Nov
  3. 2025MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators3 Apr
  4. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  6. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Service19 Jul
  7. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan

Other references · 13

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  10. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  11. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  12. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
1 more
  1. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  2. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
  3. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  4. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
  5. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  6. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
  7. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  8. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference