mlsanityData to 14 Oct 2026

Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference

arXiv 2412.00099 ↗First posted 27 Nov 2024cs.LG cs.AI cs.ARcited by 14

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. However, deploying MoEs on memory-constrained devices remains challenging, particularly when generating tokens sequentially with a batch size of one, as opposed to typical high-throughput settings involving long sequences or large batches. In this work, we optimize MoE on memory-constrained devices where only a subset of expert weights fit in DRAM. We introduce a novel cache-aware routing strategy that leverages expert reuse during token generation to improve cache locality. We evaluate our approach on language modeling, MMLU, and GSM8K benchmarks and present on-device results demonstrating 2× speedups on mobile devices, offering a flexible, training-free solution to extend MoE's applicability across real-world applications.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 28 of its 41 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

28 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  2. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  3. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  5. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  6. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  7. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  8. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2024Mixtral of Expertscited by 1,823
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  12. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
8 more
  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  2. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  3. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  4. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  7. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  8. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 8.

14 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  2. 2025Utility-Driven Speculative Decoding for Mixture-of-Expertscited by 13
  3. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  4. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  5. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  6. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  7. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  8. 2026WaterMoE: Expert-Routing-based Watermarking for High Fidelity and Efficiencycited by 1
  9. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  10. 2026ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Servingcited by 0
  11. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decodingcited by 0
  12. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
2 more
  1. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivationcited by 0
  2. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MobileMoE: Scaling On-Device Mixture of Experts
  2. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  3. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
  4. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
  5. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
  6. 2026Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
  7. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
  8. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling