mlsanityData to 7 Oct 2026

AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

arXiv 2607.20498 ↗First posted 16 Jun 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-horizon tasks. Current tool-using benchmarks for information seeking on academic graphs rely on synthetic templates, simplified solution spaces, or narrow tasks such as paper-centric tasks, leaving key challenges underexplored - realistic user intent, complex multi-step API planning, rich parameter filling for APIs, grounded answers with references, and comprehensive evaluation of both the process and the outcome. We introduce AISE-Bench, a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs. AISE-Bench release contains 1,133 QA pairs, including query taxonomies, full API execution trajectories, validated parameters, and source-grounded answers with reference links. To support high-quality annotation, we design a customized agent workflow to enable annotators to plan, execute, and revise complex API workflows efficiently. We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success. Among the 14 evaluated methods, even the strongest model (PLAY2PROMPT with Gemini-3-Pro) achieves only moderate performance and often struggles with API planning and execution. AISE-Bench establishes a challenging new testbed for quantitatively evaluating and improving the stepwise correctness, grounded summarization, and traceable reasoning of multi-step API-using LLM agents. Our code and data are available at https://aise-bench.github.io/.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

30 of 41 references.

References · 30

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  9. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  10. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  11. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  12. 2024Tool Learning with Large Language Models: A Surveycited by 221
18 more
  1. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 166
  2. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  3. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  4. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  5. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  6. 2024OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMscited by 64
  7. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 53
  8. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  9. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  10. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  11. 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
  12. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  13. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  14. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 21
  15. 2024OAG-Bench: A Human-Curated Benchmark for Academic Graph Miningcited by 14
  16. 2025PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Playcited by 14
  17. 2024SoAy: A Solution-based LLM API-using Methodology for Academic Information Seekingcited by 7
  18. 2026RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehensioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025WideSearch: Benchmarking Agentic Broad Info-Seeking
  2. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation
  3. 2026ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments
  4. 2026SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
  5. 2026SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
  6. 2026LiveNewsBench: Evaluating LLM Web Search Capabilities with Freshly Curated News
  7. 2026DailyReport: An Open-ended Benchmark for Evaluating Search Agents on Daily Search Tasks
  8. 2026K-Bench: measuring model performance on real scientific agent requests