mlsanityData to 7 Oct 2026

Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

arXiv 2603.18897 ↗First posted 19 Mar 2026cs.DC cs.AIcited by 16

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM-powered agents execute tasks through a sequential loop of model generation and tool execution. Today's serving systems serialize this loop, leaving tool latency exposed on the task critical path. This paper presents PASTE, a tool-aware agent-serving system that predicts concrete future tool invocations from recurring agent patterns and executes them speculatively while the LLM is still generating. PASTE isolates speculative results until confirmed by the LLM and jointly schedules tool execution and returning LLM sessions to avoid shifting bottlenecks to the GPU. Across deep research, coding, and scientific-agent workloads, PASTE reduces average task completion time by 43.5% and lowers observed tool latency by 1.8x.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 51 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

30 of 51 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System14 Feb
  2. 2026ToolCaching: Towards Efficient Caching for LLM Tool-calling20 Jan
  3. 2026MegaFlow: Large-Scale Distributed Orchestration System for the Agentic Era12 Jan
  4. 2025Optimizing Agentic Language Model Inference via Speculative Tool Calls17 Dec
  5. 2025Reducing Latency of LLM Search Agent via Speculation-based Algorithm-System Co-Design25 Nov
  6. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live4 Nov
  7. 2025Speculative Actions: A Lossless Framework for Faster Agentic Systems5 Oct
  8. 2025Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestration14 Sep
  9. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows10 Jul
  10. 2025Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents10 Apr

Other references · 20

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  4. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 236
  5. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
  6. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  7. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  8. 2020Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Providercited by 65
  9. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 64
  10. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 64
  11. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  12. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 49
8 more
  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  2. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  3. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  4. 2025Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AIcited by 6
  5. 2024FaaSTube: Optimizing GPU-oriented Data Transfer for Serverless Computingcited by 2
  6. 2025Databelt: A Continuous Data Path for Serverless Workflows in the 3D Compute Continuumcited by 2
  7. 2025AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Servingcited by 2
  8. 2025CausalMesh: A Formally Verified Causally Consistent Distributed Cache with Support for Client Migrationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  2. 2026Dive into Claude Code: The Design Space of Today's and Future AI Agent Systemscited by 33
  3. 2026HARBOR: Automated Harness Optimizationcited by 9
  4. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  5. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  6. 2026TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflowscited by 3
  7. 2026B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource-Constrained LLM Agentscited by 1
  8. 2026SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agentscited by 1
  9. 2026Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RLcited by 1
  10. 2026AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Servingcited by 1
  11. 2026Learning Agent Execution for KV-Cache Management in Agentic Servingcited by 1
  12. 2026Never Stop Thinking: Continuous-Time Language Agentscited by 1
4 more
  1. 2026Ghost Tool Calls: Issue-Time Privacy for Speculative Agent Toolscited by 0
  2. 2026REVISE: Validity-Guided Recovery for Online Revisions in Agent Workflowscited by 0
  3. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0
  4. 2026TomasuLLM: Out-of-Order Speculative Execution for LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents
  2. 2026Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems
  3. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
  4. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
  5. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
  6. 2026SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference
  7. 2025Dynamic System Instructions and Tool Exposure for Efficient Agentic LLMs
  8. 2026Speculative Macro Commit for Faster Tool-Using Agents