mlsanityData to 14 Oct 2026

OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading

arXiv 2609.33385 ↗First posted 27 Sep 2026cs.DC cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Semi-autoregressive diffusion large language models (dLLMs) improve decoding parallelism through iterative block-wise denoising, but scaling them with mixture-of-experts (MoE) layers introduces a large expert parameter footprint that exceeds memory-constrained GPU capacity. Expert offloading is a natural remedy, yet existing MoE serving systems target autoregressive decoding and rely on intra-iteration layer-wise prefetching: while computing one layer, they predict and load experts for subsequent layers. Under dLLM inference, block-wise routing expands the active expert working set within each iteration, making such prefetches difficult to complete in time and costly when mispredicted. Consequently, existing prefetch-based solutions often degenerate into on-demand expert loading with high decoding latency. We propose OLED-MoE, an expert offloading system that shifts the optimization target from intra-iteration prefetching to inter-iteration expert retention. Its key insight is that adjacent denoising iterations exhibit strong expert routing overlap, and token confidence indicates which experts are likely to be reused. OLED-MoE uses confidence-guided inter-iteration prediction to retain high-value experts in GPU memory without introducing extra prefetch traffic. It further compensates unavoidable cache misses through CPU-GPU cooperative execution, jointly considering dynamic expert computation load and predicted future reuse. Across diverse dLLM workloads, OLED-MoE reduces time per output token (TPOT) by 1.23x-7.93x and improves expert cache utilization by 1.44x-4.23x over state-of-the-art offloading systems. Notably, OLED-MoE approaches full-residency performance while using only 40% of the expert GPU memory, incurring merely 23% higher TPOT despite a 60% reduction in expert memory footprint. OLED-MoE's source code is publicly available at https://github.com/flashserve/OLED-MoE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 47 of its 68 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

47 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs3 Feb
  2. 2026Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs31 Jan
  3. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing21 Dec
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference3 Dec
  5. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading8 Nov
  6. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance9 Sep
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling23 Oct
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan

Other references · 39

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2025Large Language Diffusion Modelscited by 912
  7. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  8. 2024DeepSeek-V3 Technical Reportcited by 811
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  12. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
27 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  2. 2026Qwen3.5-Omni Technical Reportcited by 174
  3. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  4. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  5. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  6. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 135
  7. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  8. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  9. 2025A Survey on Diffusion Language Modelscited by 78
  10. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  11. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  12. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  13. 2024Autoregressive Models in Vision: A Surveycited by 39
  14. 2025dInfer: An Efficient Inference Framework for Diffusion Language Modelscited by 37
  15. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  16. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  17. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  18. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 21
  19. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Modelscited by 13
  20. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 13
  21. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  22. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
  23. 2025Improving the Throughput of Diffusion-based Large Language Models via a Training-Free Confidence-Aware Calibrationcited by 7
  24. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 7
  25. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generationcited by 6
  26. 2025OverFill: Two-Stage Models for Efficient Language Model Decodingcited by 2
  27. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  2. 2026TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
  4. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  5. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  7. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  8. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication