mlsanityData to 14 Oct 2026

ACEBench: Who Wins the Match Point in Tool Usage?

arXiv 2501.12851 ↗First posted 22 Jan 2025cs.CLcited by 71

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant potential in decision-making and reasoning, particularly when integrated with various tools to effectively solve complex problems. However, existing benchmarks for evaluating LLMs' tool usage face several limitations: (1) limited evaluation scenarios, often lacking assessments in real multi-turn dialogue contexts; (2) narrow evaluation dimensions, with insufficient detailed assessments of how LLMs use tools; and (3) reliance on LLMs or real API executions for evaluation, which introduces significant overhead. To address these challenges, we introduce ACEBench, a comprehensive benchmark for assessing tool usage in LLMs. ACEBench categorizes data into three primary types based on evaluation methodology: Normal, Special, and Agent. "Normal" evaluates tool usage in basic scenarios; "Special" evaluates tool usage in situations with ambiguous or incomplete instructions; "Agent" evaluates tool usage through multi-agent interactions to simulate real-world, multi-turn dialogues. We conducted extensive experiments using ACEBench, analyzing various LLMs in-depth and providing a more granular examination of error causes across different data types.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 24 of its 36 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

24 of 36 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesis24 Oct
  2. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Masking6 Oct
  3. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  4. 2024Learning to Ask: When LLM Agents Meet Unclear Instruction31 Aug
  5. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  6. 2024Tool Learning with Large Language Models: A Survey28 May
  7. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  9. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  8. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  9. 2023Augmented Language Models: a Surveycited by 403
  10. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  11. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
3 more
  1. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoningcited by 34
  2. 2022Chaining Simultaneous Thoughts for Numerical Reasoningcited by 14
  3. 2024MathViz-E: A Case-study in Domain-Specialized Tool-Using Agentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 44.

71 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025Memp: Exploring Agent Procedural Memorycited by 103
  3. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 85
  4. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  5. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  6. 2025Towards General Agentic Intelligence via Environment Scalingcited by 54
  7. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 46
  8. 2025Scaling Agents via Continual Pre-trainingcited by 40
  9. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  10. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 33
  11. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 32
  12. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?cited by 30
38 more
  1. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 23
  2. 2025A Survey on LLM Mid-Trainingcited by 21
  3. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 21
  4. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  5. 2025Sculptor: Empowering LLMs with Cognitive Agency via Active Context Managementcited by 18
  6. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 17
  7. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Usecited by 16
  8. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  9. 2026Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Textcited by 13
  10. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 12
  11. 2026Auditing Provenance Sensitivity in LLM Agent Action Selectioncited by 12
  12. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  13. 2026ERNIE 5.0 Technical Reportcited by 11
  14. 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Modelscited by 10
  15. 2026α3-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networkscited by 10
  16. 2026LOGIGEN: Logic-Driven Generation of Verifiable Agentic Taskscited by 7
  17. 2026Latent On-Policy Self-Distillationcited by 7
  18. 2025LaMDAgent: An Autonomous Framework for Post-Training Pipeline Optimization via LLM Agentscited by 6
  19. 2025Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memorycited by 6
  20. 2026Agent-Dice: Disentangling Knowledge Updates via Geometric Consensus for Agent Continual Learningcited by 6
  21. 2026AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesiscited by 6
  22. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanationcited by 5
  23. 2025SynthTools: A Framework for Scaling Synthetic Tools for Agent Developmentcited by 5
  24. 2025Evaluating Multi-Turn Bargain Skills in LLM-Based Seller Agentcited by 4
  25. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 4
  26. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 4
  27. 2025InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agentscited by 3
  28. 2026Simulating Complex Multi-Turn Tool Calling Interactions in Stateless Execution Environmentscited by 3
  29. 2026Long-Horizon Plan Execution in Large Tool Spaces through Entropy-Guided Branchingcited by 3
  30. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 3
  31. 2026ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learningcited by 3
  32. 2025NES: An Instruction-Free, Low-Latency Next Edit Suggestion Framework Powered by Learned Historical Editing Trajectoriescited by 2
  33. 2026ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulationcited by 2
  34. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  35. 2026AutoTool: Automatic Scaling of Tool-Use Capabilities in RL via Decoupled Entropy Constraintscited by 2
  36. 2026RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agentscited by 2
  37. 2025Instruction-Following Evaluation in Function Calling for Large Language Modelscited by 1
  38. 2026Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Callscited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  3. 2026Benchmarking LLM Tool-Use in the Wild
  4. 2023AgentBench: Evaluating LLMs as Agents
  5. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
  6. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  7. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
  8. 2025MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use