# Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

arXiv: https://arxiv.org/abs/2603.18897 · HTML: https://mlsanity.com/paper/2603.18897/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 30)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Parrot: Efficient Serving of LLM-based Applications with Semantic Variable](https://mlsanity.com/paper/2405.19888/) (2024)
- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction](https://mlsanity.com/paper/2505.23416/) (2025)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction](https://mlsanity.com/paper/2404.08509/) (2024)
- [KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse](https://mlsanity.com/paper/2502.16002/) (2025)
- [KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows](https://mlsanity.com/paper/2507.07400/) (2025)
- [MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving](https://mlsanity.com/paper/2404.02015/) (2024)
- [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](https://mlsanity.com/paper/2511.02230/) (2025)
- [Strata: Hierarchical Context Caching for Long Context Language Model Serving](https://mlsanity.com/paper/2508.18572/) (2025)
- [Speculative Actions: A Lossless Framework for Faster Agentic Systems](https://mlsanity.com/paper/2510.04371/) (2025)
- [Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents](https://mlsanity.com/paper/2504.07347/) (2025)
- [Alignment for Efficient Tool Calling of Large Language Models](https://mlsanity.com/paper/2503.06708/) (2025)
- [ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System](https://mlsanity.com/paper/2602.13692/) (2026)
- [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](https://mlsanity.com/paper/2601.22705/) (2026)
- [Optimizing Agentic Language Model Inference via Speculative Tool Calls](https://mlsanity.com/paper/2512.15834/) (2025)
- [MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era](https://mlsanity.com/paper/2601.07526/) (2026)
- [Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestration](https://mlsanity.com/paper/2509.11067/) (2025)
- [Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design](https://mlsanity.com/paper/2511.20048/) (2025)
- [Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AI](https://mlsanity.com/paper/2509.20175/) (2025)
- [FaaSTube: Optimizing GPU-oriented Data Transfer for Serverless Computing](https://mlsanity.com/paper/2411.01830/) (2024)
- [Databelt: A Continuous Data Path for Serverless Workflows in the 3D Compute Continuum](https://mlsanity.com/paper/2508.15351/) (2025)
- [AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving](https://mlsanity.com/paper/2512.04013/) (2025)
- [ToolCaching: Towards Efficient Caching for LLM Tool-calling](https://mlsanity.com/paper/2601.15335/) (2026)
- [CausalMesh: A Formally Verified Causally Consistent Distributed Cache with Support for Client Migration](https://mlsanity.com/paper/2508.15647/) (2025)

## Built on by (most cited first, 20 of 20)

- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems](https://mlsanity.com/paper/2604.14228/) (2026)
- [HARBOR: Automated Harness Optimization](https://mlsanity.com/paper/2604.20938/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows](https://mlsanity.com/paper/2608.02680/) (2026)
- [Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management](https://mlsanity.com/paper/2605.06472/) (2026)
- [AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving](https://mlsanity.com/paper/2608.00881/) (2026)
- [SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents](https://mlsanity.com/paper/2605.21965/) (2026)
- [Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL](https://mlsanity.com/paper/2607.25816/) (2026)
- [Learning Agent Execution for KV-Cache Management in Agentic Serving](https://mlsanity.com/paper/2608.14624/) (2026)
- [B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource-Constrained LLM Agents](https://mlsanity.com/paper/2604.16469/) (2026)
- [Never Stop Thinking: Continuous-Time Language Agents](https://mlsanity.com/paper/2609.17416/) (2026)
- [Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Tools](https://mlsanity.com/paper/2606.02483/) (2026)
- [REVISE: Validity-Guided Recovery for Online Revisions in Agent Workflows](https://mlsanity.com/paper/2609.00643/) (2026)
- [When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems](https://mlsanity.com/paper/2609.07741/) (2026)
- [DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving](https://mlsanity.com/paper/2609.31047/) (2026)
- [TomasuLLM: Out-of-Order Speculative Execution for LLM Agents](https://mlsanity.com/paper/2609.38201/) (2026)
- [LEAP: Learning Efficient Action Proposals For LLM Agents](https://mlsanity.com/paper/2610.02670/) (2026)
- [ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation](https://mlsanity.com/paper/2610.15246/) (2026)
- [SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decoding](https://mlsanity.com/paper/2610.17436/) (2026)

## Abstract (arXiv)

LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.