mlsanityData to 14 Oct 2026

When2Call: When (not) to Call Tools

arXiv 2504.18851 ↗First posted 26 Apr 2025cs.CLcited by 42

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling -- whether the correct tool is called with the correct parameters -- and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can't be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts at https://github.com/NVIDIA/When2Call.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 19 of its 30 references that are in the map. First posted on arXiv 26 Apr 2025.

Builds on

19 of 30 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks27 Jun
  2. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  3. 2024Tool Learning with Large Language Models: A Survey28 May
  4. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs23 Feb
  5. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  7. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  9. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  10. 2023Gorilla: Large Language Model Connected with Massive APIs24 May

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  6. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  7. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  8. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  9. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 35.

42 papers.

Sort:

By area:

  1. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  2. 2025Structured Uncertainty guided Clarification for LLM Agentscited by 41
  3. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 9
  4. 2025Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perceptioncited by 9
  5. 2026To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Callingcited by 7
  6. 2026Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Usecited by 7
  7. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  8. 2026From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agentscited by 5
  9. 2026Agentic Tool Use in Large Language Modelscited by 4
  10. 2026Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learningcited by 4
  11. 2026When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillationcited by 4
  12. 2026Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agentscited by 4
30 more
  1. 2025RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systemscited by 3
  2. 2026FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrievalcited by 3
  3. 2026Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocationscited by 3
  4. 2026To Call or Not to Call: Diagnosing Intrinsic Over-Calling Bias in LLM Agentscited by 3
  5. 2026When Should LLMs Search? Counterfactual Supervision for Search Routingcited by 3
  6. 2026Therefore I am. I Thinkcited by 2
  7. 2026Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learningcited by 2
  8. 2026Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Makingcited by 2
  9. 2026OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modelingcited by 2
  10. 2026Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Callingcited by 2
  11. 2025RRTL: Red Teaming Reasoning Large Language Models in Tool Learningcited by 1
  12. 2026ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialoguescited by 1
  13. 2026Beyond Text-to-SQL: An Agentic LLM System for Governed Enterprise Analytics APIscited by 1
  14. 2026Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environmentscited by 1
  15. 2026Strategic Decision Support for AI Agentscited by 1
  16. 2026SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routingcited by 1
  17. 2026SearchJev: A Fast and Calibrated System-1 Model for Search Agentscited by 1
  18. 2026AutoAdapt: An Automated Domain Adaptation Framework for LLMscited by 0
  19. 2026Mi:dm K 2.5 Procited by 0
  20. 2026Task-Aware Calibration: Provably Optimal Decoding in LLMscited by 0
  21. 2026When History Lies: Evaluating and Improving Tool Use under Misleading Multi-Turn Historiescited by 0
  22. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  23. 2026Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Returncited by 0
  24. 2026Can One Adapted Model Do It All? Fine-Tuning Strategy Selection for Customer Support LLMscited by 0
  25. 2026When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMscited by 0
  26. 2026DelegationBench: Measuring When AI Agents Should Ask Before Actingcited by 0
  27. 2026SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usabilitycited by 0
  28. 2026Sensitivity or Criterion? Measuring and Moving When Language Models Actcited by 0
  29. 2026Towards Universal Calibration for In-Context Classificationcited by 0
  30. 2026Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loopscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Tool Calling in LLMs with the International Tool Calling Dataset
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Tools
  3. 2026Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
  4. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  5. 2025Tool Calling for Arabic LLMs: Data Strategies and Instruction Tuning
  6. 2024Learning to Ask: When LLM Agents Meet Unclear Instruction
  7. 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models
  8. 2026Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation