mlsanityData to 13 Oct 2026

OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

arXiv 2609.14237 ↗First posted 13 Sep 2026cs.DC cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execution during decode. This operator-level disaggregated serving (ODS) can improve hardware matching and enable independent scaling, particularly across heterogeneous devices. However, existing systems fix operator boundaries and lack a unified characterization of when disaggregation reduces serving cost. We present OpWeave, an end-to-end framework for heterogeneous ODS. OpWeave provides an analytical cost model that bounds the gains of homogeneous and heterogeneous ODS over colocated serving. It jointly optimizes operator partitioning and deployment configuration through a regularity-aware planner that keeps the search tractable even for hybrid-attention models. A vLLM-based runtime executes the synthesized plans with flexible operator stages across heterogeneous device groups. In our evaluation, OpWeave reduces serving cost by up to 1.78× on homogeneous and 1.89× on heterogeneous GPU clusters relative to the best feasible baseline, while meeting latency SLOs.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 54 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

30 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems10 Feb

Other references · 29

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,117
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 990
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 602
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 545
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 514
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 361
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 194
  11. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  12. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 88
17 more
  1. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  2. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  3. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  4. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 37
  5. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  6. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  7. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  8. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  9. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  10. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  11. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 18
  12. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  13. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devicescited by 11
  14. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9
  15. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  16. 2026Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloadscited by 2
  17. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Scheduling
  2. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  3. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  4. 2026LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
  5. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
  6. 2026When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
  7. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  8. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure