mlsanityData to 7 Oct 2026

Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate

arXiv 2502.12224 ↗First posted 17 Feb 2025cs.AI cs.LGcited by 13

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, and their application in edge scenarios has attracted significant attention. However, sparse-activated Mixture-of-Experts (MoE) models, which are well suited for edge scenarios, have received relatively little attention due to their high memory demands. Offload-based methods have been proposed to address this challenge, but they face difficulties with expert prediction. Inaccurate expert predictions can result in prolonged inference delays. To promote the application of MoE models in edge scenarios, we propose Fate, an offloading system designed for MoE models to enable efficient inference in resource-constrained environments. The key insight behind Fate is that gate inputs from adjacent layers can be effectively used for expert prefetching, achieving high prediction accuracy without additional GPU overhead. Furthermore, Fate employs a shallow-favoring expert caching strategy that increases the expert hit rate to 99%. Additionally, Fate integrates tailored quantization strategies for cache optimization and IO efficiency. Experimental results show that, compared to Load on Demand and Expert Activation Path-based method, Fate achieves up to 4.5x and 1.9x speedups in prefill speed and up to 4.1x and 2.2x speedups in decoding speed, respectively, while maintaining inference quality. Moreover, Fate's performance improvements are scalable across different memory budgets.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 46 of its 64 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

46 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  4. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  5. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2024Qwen2 Technical Reportcited by 2,076
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2024Mixtral of Expertscited by 1,785
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
28 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2022Planning-oriented Autonomous Drivingcited by 1,068
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  6. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  9. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  13. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  14. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  15. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  16. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  17. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  18. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  19. 2023Large Language Models for Robotics: A Surveycited by 132
  20. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  21. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  22. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  23. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  24. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  25. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  26. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  27. 2024Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Networkcited by 16
  28. 2024A General-Purpose Device for Interaction with LLMscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

  1. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  2. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  3. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  4. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 3
  5. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inferencecited by 3
  6. 2025EC2MoE: Adaptive End-Cloud Pipeline Collaboration Enabling Scalable Mixture-of-Experts Inferencecited by 2
  7. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  8. 2026Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Expertscited by 2
  9. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  10. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  11. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  12. 2026Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimescited by 0
1 more
  1. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  2. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  3. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  5. 2025MoE2: Optimizing Collaborative Inference for Edge Large Language Models
  6. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  7. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
  8. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution