mlsanityData to 7 Oct 2026

DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs

arXiv 2602.03495 ↗First posted 3 Feb 2026cs.DC cs.LGcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture of Experts (MoE) architectures significantly enhance the capacity of LLMs without proportional increases in computation, but at the cost of a vast parameter size. Offloading MoE expert parameters to host memory and leveraging both CPU and GPU computation has recently emerged as a promising direction to support such models on resourceconstrained local PC platforms. While promising, we notice that existing approaches mismatch the dynamic nature of expert workloads, which leads to three fundamental inefficiencies: (1) Static expert assignment causes severe CPUGPU load imbalance, underutilizing CPU and GPU resources; (2) Existing prefetching techniques fail to accurately predict high-workload experts, leading to costly inaccurate prefetches; (3) GPU cache policies neglect workload dynamics, resulting in poor hit rates and limited effectiveness. To address these challenges, we propose DALI, a workloaDAware offLoadIng framework for efficient MoE inference on local PCs. To fully utilize hardware resources, DALI first dynamically assigns experts to CPU or GPU by modeling assignment as a 0-1 integer optimization problem and solving it efficiently using a Greedy Assignment strategy at runtime. To improve prefetching accuracy, we develop a Residual-Based Prefetching method leveraging inter-layer residual information to accurately predict high-workload experts. Additionally, we introduce a Workload-Aware Cache Replacement policy that exploits temporal correlation in expert activations to improve GPU cache efficiency. By evaluating across various MoE models and settings, DALI achieves significant speedups in the both prefill and decoding phases over the state-of-the-art offloading frameworks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 42 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

24 of 42 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  2. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  3. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  5. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  6. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design24 Oct
  7. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  8. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  11. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  7. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  12. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devicescited by 0
  2. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  2. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  3. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  4. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  5. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
  6. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts
  7. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  8. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution