# AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

arXiv: https://arxiv.org/abs/2607.20498 · HTML: https://mlsanity.com/paper/2607.20498/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [User Behavior Simulation with Large Language Model based Agents](https://mlsanity.com/paper/2306.02552/) (2023)
- [ToolGen: Unified Tool Retrieval and Calling via Generation](https://mlsanity.com/paper/2410.03439/) (2024)
- [AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning](https://mlsanity.com/paper/2406.11200/) (2024)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL](https://mlsanity.com/paper/2509.10446/) (2025)
- [Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks](https://mlsanity.com/paper/2407.00121/) (2024)
- [DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literature](https://mlsanity.com/paper/2405.04819/) (2024)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents](https://mlsanity.com/paper/2509.18119/) (2025)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks](https://mlsanity.com/paper/2509.01396/) (2025)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining](https://mlsanity.com/paper/2402.15810/) (2024)
- [PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Play](https://mlsanity.com/paper/2503.14432/) (2025)
- [SoAy: A Solution-based LLM API-using Methodology for Academic Information Seeking](https://mlsanity.com/paper/2405.15165/) (2024)
- [RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension](https://mlsanity.com/paper/2601.14289/) (2026)

## Abstract (arXiv)

Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-horizon tasks. Current tool-using benchmarks for information seeking on academic graphs rely on synthetic templates, simplified solution spaces, or narrow tasks such as paper-centric tasks, leaving key challenges underexplored - realistic user intent, complex multi-step API planning, rich parameter filling for APIs, grounded answers with references, and comprehensive evaluation of both the process and the outcome. We introduce AISE-Bench, a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs. AISE-Bench release contains 1,133 QA pairs, including query taxonomies, full API execution trajectories, validated parameters, and source-grounded answers with reference links. To support high-quality annotation, we design a customized agent workflow to enable annotators to plan, execute, and revise complex API workflows efficiently. We develop a comprehensive evaluation protocol measuring answer quality, reference grounding, API-planning correctness, and execution success. Among the 14 evaluated methods, even the strongest model (PLAY2PROMPT with Gemini-3-Pro) achieves only moderate performance and often struggles with API planning and execution. AISE-Bench establishes a challenging new testbed for quantitatively evaluating and improving the stepwise correctness, grounded summarization, and traceable reasoning of multi-step API-using LLM agents. Our code and data are available at https://aise-bench.github.io/.