mlsanityData to 7 Oct 2026

Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution

arXiv 2406.00059 ↗First posted 29 May 2024cs.CL cs.DC cs.LGcited by 16

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The complexity of large language model (LLM) serving workloads has substantially increased due to the integration with external tool invocations, such as ChatGPT plugins. In this paper, we identify a new opportunity for efficient LLM serving for requests that trigger tools: tool partial execution alongside LLM decoding. To this end, we design Conveyor, an efficient LLM serving system optimized for handling requests involving external tools. We introduce a novel interface for tool developers to expose partial execution opportunities to the LLM serving system and a request scheduler that facilitates partial tool execution. Our results demonstrate that tool partial execution can improve request completion latency by up to 38.8%.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 28 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

20 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inference2 Feb

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  11. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  12. 2023An LLM Compiler for Parallel Function Callingcited by 170
7 more
  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  2. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  3. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  4. 2022Validating Large Language Models with ReLMcited by 33
  5. 2024Optimizing LLM Queries in Relational Data Analytics Workloadscited by 20
  6. 2023Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Modelscited by 10
  7. 2024Plato: Plan to Efficiently Decode for Large Language Model Inferencecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 9.

16 papers.

Sort:

By area:

  1. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  2. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 35
  3. 2024Asynchronous LLM Function Callingcited by 18
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  5. 2025Sherlock: Reliable and Efficient Agentic Workflow Executioncited by 10
  6. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
  7. 2025PICE: A Semantic-Driven Progressive Inference System for LLM Serving in Cloud-Edge Networkscited by 5
  8. 2025Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agentscited by 5
  9. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
  10. 2026ToolSpec: Accelerating Tool Calling via Schema-Aware and Retrieval-Augmented Speculative Decodingcited by 4
  11. 2025Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspectivecited by 3
  12. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
4 more
  1. 202698× Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Routercited by 2
  2. 2026ToolCaching: Towards Efficient Caching for LLM Tool-callingcited by 1
  3. 2026Second Thought: Reasoning in Parallel as LLM Agents Act and Observecited by 0
  4. 2026Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  2. 2024Achieving Tool Calling Functionality in LLMs Using Only Prompt Engineering Without Fine-Tuning
  3. 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
  4. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
  5. 2025Unified Tool Integration for LLMs: A Protocol-Agnostic Approach to Function Calling
  6. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Information
  7. 2024Less is More: Optimizing Function Calling for LLM Execution on Edge Devices
  8. 2026LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs