mlsanityData to 14 Oct 2026

Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment

arXiv 2609.07370 ↗First posted 7 Sep 2026cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 16 of its 24 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

16 of 24 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025On the Robustness of Agentic Function Calling1 Apr
  2. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Masking6 Oct
  3. 2024Octopus: On-device language model for function calling of software APIs2 Apr
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 10

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  7. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  8. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  9. 2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
  10. 2025Small Language Models for Efficient Agentic Tool Calling: Outperforming Large Models with Targeted Fine-tuningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
  2. 2024Less is More: Optimizing Function Calling for LLM Execution on Edge Devices
  3. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models
  4. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
  5. 2025Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge
  6. 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices
  7. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
  8. 2025EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model