# An LLM-Tool Compiler for Fused Parallel Function Calling

arXiv: https://arxiv.org/abs/2405.17438 · HTML: https://mlsanity.com/paper/2405.17438/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 23 of 23)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation](https://mlsanity.com/paper/2307.15337/) (2023)
- [Extending Context Window of Large Language Models via Semantic Compression](https://mlsanity.com/paper/2312.09571/) (2023)
- [GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots](https://mlsanity.com/paper/2404.15500/) (2024)
- [APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding](https://mlsanity.com/paper/2401.06761/) (2024)
- [Efficient Prompting via Dynamic In-Context Learning](https://mlsanity.com/paper/2305.11170/) (2023)
- [Evaluating Tool-Augmented Agents in Remote Sensing Platforms](https://mlsanity.com/paper/2405.00709/) (2024)
- [OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload](https://mlsanity.com/paper/2307.03290/) (2023)
- [GeckOpt: LLM System Efficiency via Intent-Based Tool Selection](https://mlsanity.com/paper/2404.15804/) (2024)
- [Hardware-Aware Machine Learning: Modeling and Optimization](https://mlsanity.com/paper/1809.05476/) (2018)
- [Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantization](https://mlsanity.com/paper/2312.15322/) (2023)

## Built on by (most cited first, 15 of 15)

- [LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions](https://mlsanity.com/paper/2509.18970/) (2025)
- [LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios](https://mlsanity.com/paper/2508.17692/) (2025)
- [Asynchronous LLM Function Calling](https://mlsanity.com/paper/2412.07017/) (2024)
- [Toward Efficient Agents: Memory, Tool learning, and Planning](https://mlsanity.com/paper/2601.14192/) (2026)
- [Optimizing Agentic Language Model Inference via Speculative Tool Calls](https://mlsanity.com/paper/2512.15834/) (2025)
- [Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference](https://mlsanity.com/paper/2501.12959/) (2025)
- [SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge Refinement](https://mlsanity.com/paper/2504.03561/) (2025)
- [LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching](https://mlsanity.com/paper/2406.06799/) (2024)
- [FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinement](https://mlsanity.com/paper/2505.20192/) (2025)
- [Optimizing Agentic Workflows using Meta-tools](https://mlsanity.com/paper/2601.22037/) (2026)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries](https://mlsanity.com/paper/2506.18728/) (2025)
- [Less is More: Optimizing Function Calling for LLM Execution on Edge Devices](https://mlsanity.com/paper/2411.15399/) (2024)
- [Mind the Gap: Time-of-Check to Time-of-Use Vulnerabilities in LLM-Enabled Agents](https://mlsanity.com/paper/2508.17155/) (2025)
- [CIPO: Counterfactual Imagination Policy Optimization for Adaptive Tool Granularity Selection](https://mlsanity.com/paper/2610.04991/) (2026)

## Abstract (arXiv)

State-of-the-art sequential reasoning in Large Language Models (LLMs) has expanded the capabilities of Copilots beyond conversational tasks to complex function calling, managing thousands of API calls. However, the tendency of compositional prompting to segment tasks into multiple steps, each requiring a round-trip to the GPT APIs, leads to increased system latency and costs. Although recent advancements in parallel function calling have improved tool execution per API call, they may necessitate more detailed in-context instructions and task breakdown at the prompt level, resulting in higher engineering and production costs. Inspired by the hardware design principles of multiply-add (MAD) operations, which fuse multiple arithmetic operations into a single task from the compiler's perspective, we propose LLM-Tool Compiler, which selectively fuses similar types of tool operations under a single function at runtime, presenting them as a unified task to the LLM. This selective fusion inherently enhances parallelization and efficiency. Benchmarked on a large-scale Copilot platform, LLM-Tool Compiler achieves up to four times more parallel calls than existing methods, reducing token costs and latency by up to 40% and 12%, respectively.