mlsanityData to 14 Oct 2026

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios

arXiv 2506.13977 ↗First posted 11 Jun 2025cs.SE cs.CLcited by 19

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The ability of large language models (LLMs) to utilize external tools has enabled them to tackle an increasingly diverse range of tasks. However, as the tasks become more complex and long-horizon, the intricate tool utilization process may trigger various unexpected errors. Therefore, how to effectively handle such errors, including identifying, diagnosing, and recovering from them, has emerged as a key research direction for advancing tool learning. In this work, we first extensively analyze the types of errors encountered during the function-calling process on several competitive tool evaluation benchmarks. Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation benchmark specialized for tool learning. Building upon a novel evolutionary strategy for dataset construction, CRITICTOOL holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios. We conduct extensive experiments on CRITICTOOL, and validate the generalization and effectiveness of our constructed benchmark strategy. We also provide an in-depth analysis of the tool reflection ability on various LLMs, offering a new perspective on the field of tool learning in LLMs. The code is available at hrefhttps://github.com/Shellorley0513/CriticToolhttps://github.com/Shellorley0513/CriticTool.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 30 of its 45 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

30 of 45 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions10 Oct
  2. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls4 Sep
  3. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  4. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks27 Jun
  5. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  6. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction11 Jan
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  8. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  9. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  7. 2024A Survey on Large Language Models for Code Generationcited by 658
  8. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  9. 2024LLM Critics Help Catch LLM Bugscited by 147
  10. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  11. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  12. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
9 more
  1. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
  2. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  3. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 75
  4. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  5. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  6. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  7. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 25
  8. 2024LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvementcited by 11
  9. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  3. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 26
  4. 2026SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agentscited by 25
  5. 2026ExpSeek: Self-Triggered Experience Seeking for Web Agentscited by 23
  6. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  7. 2025DualVLA: Building a Generalizable Embodied Agent via Partial Decoupling of Reasoning and Actioncited by 12
  8. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 11
  9. 2025Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Modelscited by 11
  10. 2026Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errorscited by 10
  11. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  12. 2026Recursive Self-Evolving Agents via Held-Out Selectioncited by 7
7 more
  1. 2026Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generationcited by 5
  2. 2026Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoningcited by 3
  3. 2026Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Callingcited by 2
  4. 2026Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Modelscited by 1
  5. 2026AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?cited by 0
  6. 2026What to Format and How: A Benchmark and Workflow Approach for Document Formattingcited by 0
  7. 2026The Backdrop Exposes What the World Around an Agent Costs Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-Evolving Critique Abilities in Large Language Models
  2. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  3. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  4. 2025Enhancing Tool Learning in Large Language Models with Hierarchical Error Checklists
  5. 2026Try, Check and Retry: A Divide-and-Conquer Framework for Boosting Long-context Tool-Calling Performance of LLMs
  6. 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques
  7. 2026CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes
  8. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarification