mlsanityData to 7 Oct 2026

Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving

arXiv 2609.27085 ↗First posted 22 Sep 2026cs.DC cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Software engineering coding agents also fits: Financial trading and forecasting agents

also in 1 more
Abstract (arXiv)

As serving capacity demand surpasses that of training, serving efficiency becomes increasingly important. Prefill-decode (P/D) disaggregation improves serving efficiency through specialization and isolation of the two phases. These benefits rest on a static partitioning. Phase demand, however, is not static. We observe that in a large LLM fleet the ratio of uncached input to output tokens has peak-to-mean ratios up to 4.7x at minute timescales, and that in a public agentic trace the hourly ratio spans a median 24.5x within a single day, while reassigning a replica takes tens of minutes. Agentic traffic sharpens the mismatch. Sizing each pool at its ninety-fifth percentile leaves up to 17% of cluster capacity unused; sizing below it converts the same imbalance into queueing and unrealized throughput. We present Crossflow, which makes this boundary elastic without changing node roles. Each decode node publishes a short-lived, revocable lease that bounds local-prefill compute, KV capacity, transfer work, and projected output. Across public and internal traces, Crossflow improves token throughput by 16.2-17.4% on geometric mean over static P/D, and by up to 43.4% at high load, while reducing mean TTFT at every evaluated point.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 56 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

30 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Serving29 Jun
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov

Other references · 28

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
16 more
  1. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  2. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  3. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  4. 2023Stateful Large Language Model Serving with Pensievecited by 59
  5. 2024Queue management for slo-oriented large language model servingcited by 36
  6. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  7. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 21
  8. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  9. 2024RelayAttention for Efficient Large Language Model Serving with Long System Promptscited by 13
  10. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  11. 2025Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Servingcited by 12
  12. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  13. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 8
  14. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Managementcited by 7
  15. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Servingcited by 5
  16. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
  2. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
  3. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
  4. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
  5. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity
  6. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  7. 2026Cross-Model Autoscaling for Shared LLM Serving
  8. 2025EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestration