# Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution arXiv: https://arxiv.org/abs/2406.00059 · HTML: https://mlsanity.com/paper/2406.00059/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. - Builds on 20 of its 28 references that are in the map. - First posted on arXiv 29 May 2024. ## Placed in - Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/) ## Builds on (most cited first, 20 of 20) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022) - [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024) - [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024) - [ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models](https://mlsanity.com/paper/2305.18323/) (2023) - [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023) - [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024) - [Punica: Multi-Tenant LoRA Serving](https://mlsanity.com/paper/2310.18547/) (2023) - [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023) - [Validating Large Language Models with ReLM](https://mlsanity.com/paper/2211.15458/) (2022) - [InferCept: Efficient Intercept Support for Augmented Large Language Model Inference](https://mlsanity.com/paper/2402.01869/) (2024) - [Optimizing LLM Queries in Relational Data Analytics Workloads](https://mlsanity.com/paper/2403.05821/) (2024) - [Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models](https://mlsanity.com/paper/2305.17077/) (2023) - [Plato: Plan to Efficiently Decode for Large Language Model Inference](https://mlsanity.com/paper/2402.12280/) (2024) ## Built on by (most cited first, 17 of 17) - [Number Cookbook: Number Understanding of Language Models and How to Improve It](https://mlsanity.com/paper/2411.03766/) (2024) - [KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems](https://mlsanity.com/paper/2510.12872/) (2025) - [Asynchronous LLM Function Calling](https://mlsanity.com/paper/2412.07017/) (2024) - [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026) - [Sherlock: Reliable and Efficient Agentic Workflow Execution](https://mlsanity.com/paper/2511.00330/) (2025) - [Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale](https://mlsanity.com/paper/2608.00101/) (2026) - [PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networks](https://mlsanity.com/paper/2501.09367/) (2025) - [Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents](https://mlsanity.com/paper/2505.12065/) (2025) - [Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference](https://mlsanity.com/paper/2601.12967/) (2026) - [ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decoding](https://mlsanity.com/paper/2604.13519/) (2026) - [Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective](https://mlsanity.com/paper/2511.00739/) (2025) - [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026) - [ToolCaching: Towards Efficient Caching for LLM Tool-calling](https://mlsanity.com/paper/2601.15335/) (2026) - [98× Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router](https://mlsanity.com/paper/2603.12646/) (2026) - [Second Thought: Reasoning in Parallel as LLM Agents Act and Observe](https://mlsanity.com/paper/2608.13667/) (2026) - [Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It](https://mlsanity.com/paper/2609.18849/) (2026) - [RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution](https://mlsanity.com/paper/2610.14659/) (2026) ## Abstract (arXiv) The complexity of large language model (LLM) serving workloads has substantially increased due to the integration with external tool invocations, such as ChatGPT plugins. In this paper, we identify a new opportunity for efficient LLM serving for requests that trigger tools: tool partial execution alongside LLM decoding. To this end, we design Conveyor, an efficient LLM serving system optimized for handling requests involving external tools. We introduce a novel interface for tool developers to expose partial execution opportunities to the LLM serving system and a request scheduler that facilitates partial tool execution. Our results demonstrate that tool partial execution can improve request completion latency by up to 38.8%.