# Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents

arXiv: https://arxiv.org/abs/2607.27083 · HTML: https://mlsanity.com/paper/2607.27083/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 26 of 26)

- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [Smart "Predict, then Optimize"](https://mlsanity.com/paper/1710.08005/) (2017)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting](https://mlsanity.com/paper/2405.00823/) (2024)
- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [Budget-Aware Tool Use Enables Effective Agent Scaling](https://mlsanity.com/paper/2511.17006/) (2025)
- [When2Call: When (not) to Call Tools](https://mlsanity.com/paper/2504.18851/) (2025)
- [CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents](https://mlsanity.com/paper/2511.02734/) (2025)
- [LLM Agents Already Know When to Call Tools -- Even Without Reasoning](https://mlsanity.com/paper/2605.09252/) (2026)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning](https://mlsanity.com/paper/2411.16313/) (2024)
- [Pandora's Box Problem with Order Constraints](https://mlsanity.com/paper/2002.06968/) (2020)
- [Choppy: Cut Transformer For Ranked List Truncation](https://mlsanity.com/paper/2004.13012/) (2020)
- [Budget-Constrained Tool Learning with Planning](https://mlsanity.com/paper/2402.15960/) (2024)
- [EcoAct: Economic Agent Determines When to Register What Action](https://mlsanity.com/paper/2411.01643/) (2024)
- [To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling](https://mlsanity.com/paper/2605.00737/) (2026)
- [Recent Developments in Pandora's Box Problem: Variants and Applications](https://mlsanity.com/paper/2308.12242/) (2023)
- [TASR: Training-Free Adaptive Stopping for Iterative Retrieval](https://mlsanity.com/paper/2606.13814/) (2026)
- [Utility-Guided Agent Orchestration for Efficient LLM Tool Use](https://mlsanity.com/paper/2603.19896/) (2026)
- [Online Pandora's Box for Contextual LLM Cascading](https://mlsanity.com/paper/2606.07392/) (2026)

## Built on by (most cited first, 4 of 4)

- [What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review](https://mlsanity.com/paper/2609.33153/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)
- [CASE: Cost-Aware Stopping for Efficient Long-Video Agents](https://mlsanity.com/paper/2610.05400/) (2026)
- [Agentic RCA for Internet-Scale Services Using Constrained Creativity](https://mlsanity.com/paper/2610.08622/) (2026)

## Abstract (arXiv)

As LLM agents increasingly depend on diverse external services such as search engines, databases, and connectors, agent harnesses face a fundamental tool-selection challenge: acquiring too few tools leaves the task under-informed, while too many adds cost, context load, and privacy exposure. Routers and retrievers can rank candidate tools by relevance, but a ranking alone does not determine how many are worth selecting. Existing approaches leave acquisition under heterogeneous costs unaddressed. We formulate this decision as cost-aware marginal decision-focused stopping (CAM-DF) over ranked tool prefixes, with CAM-DF-lite as a compact interpretable variant. We train directly on the offline gap between stopping now and the best continuation: its sign labels the decision, its magnitude weights each error by the payoff at stake. We prove this objective is Bayes-aligned with the stopping target and that score-only rules are suboptimal under heterogeneous costs. We evaluate on 1,343 tasks across five tool-use domains. On τ-bench Retail, CAM-DF attains the highest payoff among deployable methods, with gains over a predict-then-threshold baseline across all five ranking sources and two cost regimes. Our approach is state-of-the-art under heterogeneous costs and high cost pressure, with larger gains under weaker rankings. In live execution, CAM-DF exposes the agent to 37% fewer tools than full access while maintaining comparable task success. The CAM-DF family is a lightweight pre-execution plugin that turns existing tool rankings into lower-cost acquisition decisions without fine-tuning the underlying LLM.