mlsanityData to 14 Oct 2026

PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving

arXiv 2609.16491 ↗First posted 15 Sep 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Efficient reasoning and agent inference

Abstract (arXiv)

LLM agents execute long-horizon workflows where each model response determines the progress of subsequent tool interactions and environment transitions. Unlike chatbot serving, where TTFT and TPOT SLO constraints are critical, agentic workloads are completion-oriented and increasingly governed by job completion time (JCT). This shift challenges existing LLM serving designs optimized around token SLOs. Through a systematic exploration of scheduling and parallelism, we uncover a previously overlooked principle for agent serving: JCT is governed by the balance between prefill and decode efficiency. A prefill-prioritized scheduling policy achieves the best TTFT and the highest decode throughput, yet fails to attain the lowest JCT. This principle further reshapes the parallelism landscape: we show that pipeline parallelism (PP), long overlooked because it offers little decode-latency advantage, can reduce JCT by providing a more favorable balance between prefill and decode efficiency. Based on these insights, we build PipeSwift, an optimized open-source pipeline-parallel runtime integrated with a tailored micro-batch partitioning strategy co-designed with schedule considering the above trade-off, and pipeline-integrated multi-token prediction. Evaluated on real coding and web-search agent trajectories with two 360B+ MoE models on 64 H800 GPUs, PipeSwift reduces overall JCT by up to 1.45× over SGLang wide-EP, 2.33× over vLLM PP2, and 1.54× over today's state-of-the-art open-source PD-disaggregated deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient reasoning and agent inference, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 38 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

28 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System14 Feb
  2. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  3. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb

Other references · 25

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
13 more
  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  2. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  3. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instancescited by 87
  4. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  5. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  6. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  7. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  8. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inferencecited by 15
  9. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  10. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  11. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  12. 2026ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RLcited by 8
  13. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Servingcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
  2. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling
  3. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
  4. 2026AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
  5. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  6. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering
  7. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
  8. 2025Astraea: A State-Aware Scheduling Engine for LLM-Powered Agents