mlsanityData to 14 Oct 2026

MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems

arXiv 2505.11415 ↗First posted 16 May 2025cs.LG cs.DCcited by 11

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The sparse Mixture-of-Experts (MoE) architecture is increasingly favored for scaling Large Language Models (LLMs) efficiently, but it depends on heterogeneous compute and memory resources. These factors jointly affect system Cost, Accuracy, and Performance (CAP), making trade-offs inevitable. Existing benchmarks often fail to capture these trade-offs accurately, complicating practical deployment decisions. To address this, we introduce MoE-CAP, a benchmark specifically designed for MoE systems. Our analysis reveals that achieving an optimal balance across CAP is difficult with current hardware; MoE systems typically optimize two of the three dimensions at the expense of the third-a dynamic we term the MoE-CAP trade-off. To visualize this, we propose the CAP Radar Diagram. We further introduce sparsity-aware performance metrics-Sparse Memory Bandwidth Utilization (S-MBU) and Sparse Model FLOPS Utilization (S-MFU)-to enable accurate performance benchmarking of MoE systems across diverse hardware platforms and deployment scenarios.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 33 of its 59 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

33 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  3. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct

Other references · 29

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
17 more
  1. 2019MLPerf Inference Benchmarkcited by 199
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  3. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  4. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  5. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  6. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  7. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  8. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  9. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  10. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 79
  11. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  12. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  13. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  14. 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 26
  15. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 20
  16. 2024The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Modelscited by 17
  17. 2023Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collectionscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  2. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  3. 2026Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teamscited by 2
  4. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  5. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Studycited by 1
  6. 2026Searching for Plans You Can Actually Build: A Realizability-Aware Full-Space Optimizer for MoE Training and Servingcited by 1
  7. 2026BatchGen: An Architecture for Scalable and Efficient Batch Inferencecited by 0
  8. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  9. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  10. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMscited by 0
  11. 2026GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  2. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  3. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  4. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
  5. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  7. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
  8. 2025Faster MoE LLM Inference for Extremely Large Models