mlsanityData to 14 Oct 2026

LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers

arXiv 2509.23638 ↗First posted 28 Sep 2025cs.LGcited by 8

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models face memory and PCIe latency bottlenecks when deployed on commodity hardware. Offloading expert weights to CPU memory results in PCIe transfer latency that exceeds GPU computation by several folds. We present PreScope, a prediction-driven expert scheduling system that addresses three key challenges: inaccurate activation prediction, PCIe bandwidth competition, and cross-device scheduling complexity. Our solution includes: 1) Learnable Layer-Aware Predictor (LLaPor) that captures layer-specific expert activation patterns; 2) Prefetch-Aware Cross-Layer Scheduling (PreSched) that generates globally optimal plans balancing prefetching costs and loading overhead; 3) Asynchronous I/O Optimizer (AsyncIO) that decouples I/O from computation, eliminating waiting bubbles. PreScope achieves 141% higher throughput and 74.6% lower latency than state-of-the-art solutions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 34 of its 46 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

34 of 46 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching12 Mar
  2. 2025Muon is Scalable for LLM Training24 Feb
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  6. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  7. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  8. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  9. 2024Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing25 Apr
  10. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  11. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference16 Jan
  12. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
2 more
  1. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  7. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  8. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  9. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  10. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  11. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  12. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
8 more
  1. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  2. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
  3. 2024LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inferencecited by 18
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  5. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  6. 2025LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Servingcited by 8
  7. 2025BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inferencecited by 5
  8. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  2. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  3. 2026Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUscited by 2
  4. 2026Speculating Experts Accelerates Inference for Mixture-of-Expertscited by 1
  5. 2025Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMscited by 0
  6. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  7. 2026SAEM: Stage-Aware Expert Management for Memory-Efficient MoE Inference in Chain-of-Thought Reasoningcited by 0
  8. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
  2. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  3. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  4. 2026SPICE: Speculative Prefetching with Low-Rank Expert Surrogates and Heterogeneous Orchestration for MoE Inference Acceleration
  5. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  6. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  7. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
  8. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony