mlsanityData to 7 Oct 2026

From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill

arXiv 2510.08055 ↗First posted 9 Oct 2025cs.LG cs.DCcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large Language Model (LLM) inference in production must meet stringent service-level objectives for both time-to-first-token (TTFT) and time-between-token (TBT) while maximizing throughput under fixed compute, memory, and interconnect budgets. Modern serving systems adopt stall-free scheduling techniques such as chunked prefill, which splits the processing of long prompts along the token dimension and interleaves prefill with ongoing decode iterations. While effective at stabilizing TBT, chunked prefill incurs substantial overhead in Mixture-of-Experts (MoE) models: redundant expert weight loads increase memory traffic by up to 39% and inflate energy consumption. We propose layered prefill, a new scheduling paradigm that treats transformer layer groups as the primary scheduling unit, specifically targeting MoE serving. By vertically partitioning the model into contiguous layer groups and interleaving prefill and decode across the groups, layered prefill sustains stall-free decoding while eliminating chunk-induced MoE weight reloads. It reduces off-chip bandwidth demand, lowering TTFT by up to 70%, end-to-end latency by 41% and per-token energy by up to 22%. Evaluations show that layered prefill consistently improves the TTFT--TBT Pareto frontier over chunked prefill, reducing expert-load traffic and energy cost while maintaining stall-free decoding. Overall, shifting the scheduling axis from tokens to layers unlocks a new operating regime for high-efficiency, energy-aware MoE serving in co-located environments.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 52 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

35 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts21 Jul
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2020GLU Variants Improve Transformercited by 1,724
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
21 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  9. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  11. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  12. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  13. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  14. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  15. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  16. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  17. 2024Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inferencecited by 14
  18. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  19. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspectivecited by 8
  20. 2024FlexNN: A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient Edge Devicescited by 7
  21. 2020GPU Tensor Cores for fast Arithmetic Reductionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
  2. 2026Threshold-Based Exclusive Batching for LLM Inferencecited by 0
  3. 2026P-PAS: Prefill-Pressure Adaptive Scheduling for Long-Context LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Layer Scheduling for MoE-Based LLM Reasoning
  2. 2025Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference
  3. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  4. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
  5. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
  6. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  7. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  8. 2026Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving