mlsanityData to 7 Oct 2026

DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference

arXiv 2501.10375 ↗First posted 16 Dec 2024cs.DC cs.LGcited by 14

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models, though highly effective for various machine learning tasks, face significant deployment challenges on memory-constrained devices. While GPUs offer fast inference, their limited memory compared to CPUs means not all experts can be stored on the GPU simultaneously, necessitating frequent, costly data transfers from CPU memory, often negating GPU speed advantages. To address this, we present DAOP, an on-device MoE inference engine to optimize parallel GPU-CPU execution. DAOP dynamically allocates experts between CPU and GPU based on per-sequence activation patterns, and selectively pre-calculates predicted experts on CPUs to minimize transfer latency. This approach enables efficient resource utilization across various expert cache ratios while maintaining model accuracy through a novel graceful degradation mechanism. Comprehensive evaluations across various datasets show that DAOP outperforms traditional expert caching and prefetching methods by up to 8.20x and offloading techniques by 1.35x while maintaining accuracy.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 34 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

22 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  3. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  4. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2024Mixtral of Expertscited by 1,785
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  12. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
5 more
  1. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  2. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  3. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  4. 2023Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAscited by 3
  5. 2023CRISP: Hybrid Structured Sparsity for Class-aware Model Pruningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

  1. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  2. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  3. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  4. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  5. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systemscited by 2
  6. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  7. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inferencecited by 1
  8. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devicescited by 1
  9. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  10. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  11. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systemscited by 0
  12. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
2 more
  1. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0
  2. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  2. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  3. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  4. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  5. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  6. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  7. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  8. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling