mlsanityData to 7 Oct 2026

SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration

arXiv 2412.12151 ↗First posted 11 Dec 2024cs.LG cs.AI cs.SEcited by 13

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The tool-use ability of Large Language Models (LLMs) has a profound impact on a wide range of industrial applications. However, LLMs' self-control and calibration capability in appropriately using tools remains understudied. The problem is consequential as it raises potential risks of degraded performance and poses a threat to the trustworthiness of the models. In this paper, we conduct a study on a family of state-of-the-art LLMs on three datasets with two mainstream tool-use frameworks. Our study reveals the tool-abuse behavior of LLMs, a tendency for models to misuse tools with overconfidence. We also find that this is a common issue regardless of model capability. Accordingly, we propose a novel approach, textitSMARTCAL, to mitigate the observed issues, and our results show an average of 8.6 percent increase in the QA performance and a 21.6 percent decrease in Expected Calibration Error (ECE) compared to baseline models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 30 of its 41 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

30 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Large Language Models as Tool Makers26 May
  2. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 27

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  10. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  12. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
15 more
  1. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  2. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  3. 2022Binding Language Models in Symbolic Languagescited by 224
  4. 2023Fine-tuning Language Models for Factualitycited by 219
  5. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  6. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  7. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  8. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 71
  9. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  10. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMscited by 19
  11. 2024Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Modelscited by 19
  12. 2024An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaborationcited by 16
  13. 2023Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processescited by 15
  14. 2024Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutionscited by 15
  15. 2024Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidencecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  3. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  4. 2025Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertaintycited by 16
  5. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  6. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generationcited by 9
  7. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Searchcited by 7
  8. 2025AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.orgcited by 2
  9. 2026Controlling Tool Use with Heading-Specific Activation Steeringcited by 2
  10. 2025ScheduleMe: Multi-Agent Calendar Assistantcited by 1
  11. 2026AutoSearch: Adaptive Search Depth for Efficient Agentic RAG via Reinforcement Learningcited by 1
  12. 2025LightSearcher: Efficient DeepSearch via Experiential Memorycited by 0
1 more
  1. 2026MetaRAG: Belief-Action Aligned Policy Optimization for Agentic RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs
  2. 2026The Calls are Coming from Inside the Model: Investigating Probe-based Detection of Tool-Calling Errors in LLMs
  3. 2026Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use
  4. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models
  5. 2023On the Tool Manipulation Capability of Open-source Large Language Models
  6. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
  7. 2024Tool Learning with Large Language Models: A Survey
  8. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks