# Quantitative Certification of Agentic Tool Selection

arXiv: https://arxiv.org/abs/2510.03992 · HTML: https://mlsanity.com/paper/2510.03992/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models](https://mlsanity.com/paper/2503.00564/) (2025)
- [Certified Robustness for Large Language Models with Self-Denoising](https://mlsanity.com/paper/2307.07171/) (2023)
- [CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2505.19864/) (2025)
- [Self-Organizing Agent Network for LLM-based Workflow Automation](https://mlsanity.com/paper/2508.13732/) (2025)
- [Adversarial Attacks against Neural Ranking Models via In-Context Learning](https://mlsanity.com/paper/2508.15283/) (2025)
- [Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection](https://mlsanity.com/paper/2307.16816/) (2023)
- [Certifying Knowledge Comprehension in LLMs](https://mlsanity.com/paper/2402.15929/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations](https://mlsanity.com/paper/2608.18389/) (2026)
- [Lumos: Let there be Language Model System Certification](https://mlsanity.com/paper/2512.02966/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in agentic systems, where a fundamental task is mapping user intents to relevant external tools. Errors in tool selection can have severe outcomes, such as unauthorized data access, even without modifying the agent's underlying model. Existing evaluations measure performance on curated, benign benchmarks. However, a pipeline's behavior in deployment depends on the tool pool the agent actually encounters, which in open registries is shaped by third parties. We introduce LLMCert-T, the first statistical framework that returns high-confidence upper bounds on the probability that a tool-selection pipeline satisfies a declared safety specification under a realistic tool distribution. LLMCert-T models tool-selection evaluation as a Bernoulli estimation problem, drawing inserted-tool sequences from a distribution that the safety specification fixes. To evaluate robustness against realistic deployment conditions, we instantiate this distribution as a stochastic process that generates inserted-tool sequences round by round, conditioning each round on the agent's selection in the previous round. LLMCert-T aggregates the per-trial outcomes into a one-sided Clopper-Pearson upper bound on the probability that the specification is satisfied. By returning this bound as a certificate with statistical guarantees over the inserted-tool sequence distribution, LLMCert-T makes safety claims intuitive, actionable, and comparable across models, retrievers, mitigations, and registry policies. Across popular BFCL and OpenAPI tool pools, LLMCert-T shows that current LLM agents remain fragile under Distractor Selection and Top-N Saturation specifications: their certified correctness upper bounds drop to approximately 20%, far below their clean-pool lower bounds.