SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Mixture-of-Experts (MoE) creates a structural advantage for offloading: only a small fraction of activated experts need to reside in device memory, and if they can be loaded in time for computation, offloading can in principle approach full-load performance, where all model weights reside in device memory. Yet translating MoE's structural advantage into practical offloading gains remains challenging. We propose SeqMoE to bridge this gap. To maximize expert hits, we build predictive memory management: (i) Sequence-to-sequence prediction. We are the first to recast expert activation prediction as sequence modeling, enabling accurate multi-step, multi-layer forecasts that provide a long and reliable window for downstream decisions. (ii) Joint prefetch scheduling. We formulate prefetch scheduling as Job Sequencing with Deadlines to maximize expected expert hits and improve bandwidth efficiency. (iii) Forecast-driven caching. Leveraging the recursive nature of sequence modeling, we introduce a probabilistic Belady policy for future-aware eviction. To eliminate execution bottleneck, we develop (iv) Graph-compatible offloading runtime. We derive general runtime principles encompassing compute-transparent expert placement and synchronization-free orchestration disciplines for end-to-end graph capture. With 45% expert residency, SeqMoE averages a 96.97% hit rate and 80.22% of full-load performance, advancing the state of the art in MoE offloading.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 15 of its 44 references that are in the map. First posted on arXiv 11 Sep 2026.
Builds on
15 of 44 references.
Closest ancestors · 7
Same direction, newest first
- 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference13 Jun
- 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference7 Oct
- 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
- 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models18 Dec
- 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
- 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
- 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
Other references · 8
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
- 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading
- 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
- 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers
- 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
- 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
- 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
- 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
- 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models