# TinyAgent: Function Calling at the Edge

arXiv: https://arxiv.org/abs/2409.00608 · HTML: https://mlsanity.com/paper/2409.00608/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [UFO: A UI-Focused Agent for Windows OS Interaction](https://mlsanity.com/paper/2402.07939/) (2024)
- [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://mlsanity.com/paper/2302.13007/) (2023)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [TinyGSM: achieving >80% on GSM8k with small language models](https://mlsanity.com/paper/2312.09241/) (2023)
- [LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement](https://mlsanity.com/paper/2403.15042/) (2024)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [Benchmarking Mobile Device Control Agents across Diverse Configurations](https://mlsanity.com/paper/2404.16660/) (2024)
- [ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT](https://mlsanity.com/paper/2304.14334/) (2023)

## Built on by (most cited first, 30 of 58)

- [Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks](https://mlsanity.com/paper/2503.09572/) (2025)
- [Hammer: Robust Function-Calling for On-Device Language Models via Function Masking](https://mlsanity.com/paper/2410.04587/) (2024)
- [MARFT: Multi-Agent Reinforcement Fine-Tuning](https://mlsanity.com/paper/2504.16129/) (2025)
- [Distilling LLM Agent into Small Models with Retrieval and Code Tools](https://mlsanity.com/paper/2505.17612/) (2025)
- [GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration](https://mlsanity.com/paper/2503.17709/) (2025)
- [Efficient and Scalable Estimation of Tool Representations in Vector Space](https://mlsanity.com/paper/2409.02141/) (2024)
- [LongFuncEval: Measuring the effectiveness of long context models for function calling](https://mlsanity.com/paper/2505.10570/) (2025)
- [ToolGate: Contract-Grounded and Verified Tool Execution for LLMs](https://mlsanity.com/paper/2601.04688/) (2026)
- [HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights](https://mlsanity.com/paper/2505.04846/) (2025)
- [Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC](https://mlsanity.com/paper/2506.24045/) (2025)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Automating High Energy Physics Data Analysis with LLM-Powered Agents](https://mlsanity.com/paper/2512.07785/) (2025)
- [Agentic Test-Time Scaling for WebAgents](https://mlsanity.com/paper/2602.12276/) (2026)
- [Bridging AI and Software Security: A Comparative Vulnerability Assessment of LLM Agent Deployment Paradigms](https://mlsanity.com/paper/2507.06323/) (2025)
- [Dynamic Tool Dependency Retrieval for Lightweight Function Calling](https://mlsanity.com/paper/2512.17052/) (2025)
- [Toward Super Agent System with Hybrid AI Routers](https://mlsanity.com/paper/2504.10519/) (2025)
- [DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues](https://mlsanity.com/paper/2506.22853/) (2025)
- [Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models](https://mlsanity.com/paper/2510.07248/) (2025)
- [TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices](https://mlsanity.com/paper/2511.22138/) (2025)
- [DroidCall: A Dataset for LLM-powered Android Intent Invocation](https://mlsanity.com/paper/2412.00402/) (2024)
- [Agent-X: Full Pipeline Acceleration of On-device AI Agents](https://mlsanity.com/paper/2605.10380/) (2026)
- [Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning](https://mlsanity.com/paper/2410.16029/) (2024)
- [ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Calling](https://mlsanity.com/paper/2510.14703/) (2025)
- [AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems](https://mlsanity.com/paper/2510.00229/) (2025)
- [Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)](https://mlsanity.com/paper/2501.05465/) (2025)
- [Every Software as an Agent: Blueprint and Case Study](https://mlsanity.com/paper/2502.04747/) (2025)
- [Advancing SLM Tool-Use Capability using Reinforcement Learning](https://mlsanity.com/paper/2509.04518/) (2025)
- [ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights](https://mlsanity.com/paper/2511.00074/) (2025)
- [Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMs](https://mlsanity.com/paper/2605.15077/) (2026)
- [Evoflux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents](https://mlsanity.com/paper/2606.12674/) (2026)

## Abstract (arXiv)

Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries through function calling. However, the deployment of these LLMs on the edge has not been explored since they typically require cloud-based infrastructure due to their substantial model size and computational demands. To this end, we present TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. We first show how to enable accurate function calling for open-source models via the LLMCompiler framework. We then systematically curate a high-quality dataset for function calling, which we use to fine-tune two small language models, TinyAgent-1.1B and 7B. For efficient inference, we introduce a novel tool retrieval method to reduce the input prompt length and utilize quantization to further accelerate the inference speed. As a driving application, we demonstrate a local Siri-like system for Apple's MacBook that can execute user commands through text or voice input. Our results show that our models can achieve, and even surpass, the function-calling capabilities of larger models like GPT-4-Turbo, while being fully deployed at the edge. We open-source our dataset, models, and installable package and provide a demo video for our MacBook assistant agent.