mlsanityData to 7 Oct 2026

ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario

arXiv 2501.10132 ↗First posted 17 Jan 2025cs.CLcited by 53

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at urlhttps://github.com/THUDM/ComplexFuncBench.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 14 of its 27 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

14 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning16 Oct
  2. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023Tool Learning with Foundation Models17 Apr

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  6. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  7. 2023SuperCLUE: A Comprehensive Chinese Large Language Model Benchmarkcited by 50
  8. 2024Benchmarking Agentic Workflow Generationcited by 36
  9. 2024MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 24.

53 papers.

Sort:

By area:

  1. 2025Survey on Evaluation of LLM-based Agentscited by 190
  2. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  3. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 101
  4. 2025MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Toolscited by 30
  5. 2025ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Datasetcited by 27
  6. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 21
  7. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 17
  8. 2025MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Usecited by 16
  9. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Modelscited by 16
  10. 2025Graph RAG-Tool Fusioncited by 15
  11. 2025LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queriescited by 15
  12. 2025Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systemscited by 15
38 more
  1. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 15
  2. 2025Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Callingcited by 13
  3. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  5. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?cited by 10
  6. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 8
  7. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Callingcited by 7
  8. 2025The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturationcited by 7
  9. 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Usecited by 7
  10. 2025ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactionscited by 5
  11. 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexitycited by 5
  12. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  13. 2025Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Callingcited by 4
  14. 2025Doc2Agent: Scalable Generation of Tool-Using Agents from API Documentationcited by 4
  15. 2025How Good Are LLMs at Processing Tool Outputs?cited by 4
  16. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 4
  17. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 3
  18. 2026UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agentscited by 3
  19. 2026The Scaling Laws of Skills in LLM Agent Systemscited by 3
  20. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarioscited by 3
  21. 2025Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategiescited by 2
  22. 2025ToolPRM: Fine-Grained Inference Scaling of Structured Outputs for Function Callingcited by 2
  23. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  24. 2026Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewardscited by 1
  25. 2026Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streamscited by 1
  26. 2026EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agentscited by 1
  27. 2026Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventionscited by 1
  28. 2025Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing contextcited by 0
  29. 2025ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystemcited by 0
  30. 2025From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Modelscited by 0
  31. 2026Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approachcited by 0
  32. 2026D-CORE: Incentivizing Task Decomposition in Large Reasoning Models for Complex Tool Usecited by 0
  33. 2026LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challengescited by 0
  34. 2026AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?cited by 0
  35. 2026TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Compositioncited by 0
  36. 2026Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environmentscited by 0
  37. 2026VCIFBench: Evaluating Complex Instruction Following for Video Understandingcited by 0
  38. 2026Serving Agentic Workflows with a Physical-Plan Compiler and Adaptive Runtimecited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  2. 2024Benchmarking Floworks against OpenAI & Anthropic: A Novel Framework for Enhanced LLM Function Calling
  3. 2025FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinement
  4. 2024Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation
  5. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
  6. 2025Efficient Function Orchestration for Large Language Models
  7. 2024Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks
  8. 2025STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning