mlsanityData to 7 Oct 2026

Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity

arXiv 2601.00268 ↗First posted 1 Jan 2026cs.CL cs.AIcited by 5

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

We introduce WildAGTEval, a benchmark designed to evaluate large language model (LLM) agents' function-calling capabilities under realistic API complexity. Unlike prior work that assumes an idealized API system and disregards real-world factors such as noisy API outputs, WildAGTEval accounts for two dimensions of real-world complexity: 1. API specification, which includes detailed documentation and usage constraints, and 2. API execution, which captures runtime challenges. Consequently, WildAGTEval offers (i) an API system encompassing 60 distinct complexity scenarios that can be composed into approximately 32K test configurations, and (ii) user-agent interactions for evaluating LLM agents on these scenarios. Using WildAGTEval, we systematically assess several advanced LLMs and observe that most scenarios are challenging, with irrelevant information complexity posing the greatest difficulty and reducing the performance of strong LLMs by 27.3%. Furthermore, our qualitative analysis reveals that LLMs occasionally distort user intent merely to claim task completion, critically affecting user satisfaction.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 25 of its 35 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

25 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario17 Jan
  2. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls4 Sep
  3. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?18 Jun

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  12. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
10 more
  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  3. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  4. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  5. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  6. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  7. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modelingcited by 21
  8. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  9. 2023One Size Fits All for Semantic Shifts: Adaptive Prompt Tuning for Continual Learningcited by 8
  10. 2023Adaptive Shortcut Debiasing for Online Continual Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 10
  2. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1
  3. 2026PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noisecited by 0
  4. 2026PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Callingcited by 0
  5. 2026Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  2. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  3. 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
  4. 2026UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
  5. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  6. 2025CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
  7. 2026Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Models
  8. 2024Understanding the Weakness of Large Language Model Agents within a Complex Android Environment