mlsanityData to 14 Oct 2026

Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling

arXiv 2607.25498 ↗First posted 28 Jul 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Prefill-decode disaggregation (PD) and roofline-based operator placement are common strategies for partitioning Large Language Model (LLM) inference across heterogeneous systems, but they are often insufficient in practice. End-to-end latency also depends on workload shape, runtime device contention, and persistent weight layout. We present DOPS (dynamic operator scheduling), a hardware-aware, closed-loop framework that jointly optimizes operator scheduling and blockwise weight layouts. DOPS constructs a stage-aware directed acyclic graph (DAG) and integrates two components: the Bifocal scheduler for dynamic operator-to-device placement and the Weight Layout Arbiter (WLA) for selecting hardware-efficient weight layouts under strict memory constraints. Across representative heterogeneous systems combining neural processing units (NPUs) and processing-in-memory (PIM) devices, Bifocal achieves geometric-mean speedups of 1.20× to 2.23× over the PD baseline. WLA provides an additional geometric-mean speedup of 1.28× to 1.33× over Bifocal/Linear. DOPS also supports systematic analysis of workload sensitivity and hardware scalability for LLM serving. The source code is available at https://github.com/YIAI-02/TriForm, and the visualization tool is demonstrated at https://youtu.be/YaoMCyYno0.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 83 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

36 of 83 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2025PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference11 Feb
  3. 2024Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments24 Nov
  4. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  12. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
19 more
  1. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  2. 2024Fast On-device LLM Inference with NPUscited by 62
  3. 2024NeuPIMs: NPU-PIM Heterogeneous Acceleration for Batched LLM Inferencingcited by 52
  4. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  5. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  6. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34
  7. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 30
  8. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 29
  9. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  10. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  11. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  12. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Servingcited by 19
  13. 2024Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLMcited by 18
  14. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 14
  15. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 10
  16. 2025Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMMcited by 6
  17. 2024ONNXim: A Fast, Cycle-level Multi-core NPU Simulatorcited by 5
  18. 2025Speculative Decoding for Multi-Sample Inferencecited by 5
  19. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  2. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  3. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  4. 2026When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
  5. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
  6. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  7. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  8. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs