mlsanityData to 7 Oct 2026

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

arXiv 2608.12282 ↗First posted 12 Aug 2026cs.AIcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (eValuating API and Knowledge Retrieval Agents), a benchmark of over 8,000 executable APIs across 62 domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4% on single-hop endpoint-style tasks and drops to 50--51% on compositional APIs; performance degrades by over 50% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 49 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

19 of 49 references.

References · 19

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  4. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  6. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  7. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  8. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  9. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 101
  10. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  11. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45
  12. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 21
7 more
  1. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 11
  2. 2024Multi-Document Grounded Multi-Turn Synthetic Dialog Generationcited by 7
  3. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  4. 2024NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Modelscited by 4
  5. 2025Automated Creation and Enrichment Framework for Improved Invocation of Enterprise APIs as Toolscited by 4
  6. 2025A New Query Expansion Approach via Agent-Mediated Dialogic Inquirycited by 2
  7. 2026Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Callscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
  2. 2025In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents
  3. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG
  4. 2026Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends
  5. 2025A Framework for Testing and Adapting REST APIs as LLM Tools
  6. 2026Similar Accuracy, Unequal Evidence: Search APIs as Decision Surfaces for Tool-Using Agents
  7. 2026E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
  8. 2026Long-Horizon Plan Execution in Large Tool Spaces through Entropy-Guided Branching