# ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Research

arXiv: https://arxiv.org/abs/2601.21654 · HTML: https://mlsanity.com/paper/2601.21654/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation](https://mlsanity.com/paper/2505.23885/) (2025)
- [xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations](https://mlsanity.com/paper/2506.13651/) (2025)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution](https://mlsanity.com/paper/2505.20286/) (2025)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025)
- [PaSa: An LLM Agent for Comprehensive Academic Paper Search](https://mlsanity.com/paper/2501.10120/) (2025)
- [DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL](https://mlsanity.com/paper/2509.10446/) (2025)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://mlsanity.com/paper/2407.12861/) (2024)
- [EvolveSearch: An Iterative Self-Evolving Search Agent](https://mlsanity.com/paper/2505.22501/) (2025)
- [ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks](https://mlsanity.com/paper/2508.15804/) (2025)
- [LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild](https://mlsanity.com/paper/2510.14240/) (2025)
- [DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports](https://mlsanity.com/paper/2601.08536/) (2026)
- [Step-DeepResearch Technical Report](https://mlsanity.com/paper/2512.20491/) (2025)
- [How Far Are We from Genuinely Useful Deep Research Agents?](https://mlsanity.com/paper/2512.01948/) (2025)
- [Nested Browser-Use Learning for Agentic Information Seeking](https://mlsanity.com/paper/2512.23647/) (2025)
- [WisPaper: Your AI Scholar Search Engine](https://mlsanity.com/paper/2512.06879/) (2025)
- [HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery](https://mlsanity.com/paper/2512.22899/) (2025)

## Built on by (most cited first, 3 of 3)

- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [Prompt-to-Paper: Agentic AI System for Bioinformatics](https://mlsanity.com/paper/2607.05456/) (2026)
- [Source Preference in the Wild: How LLM Agents Favor Items by Source, and How to Reduce It](https://mlsanity.com/paper/2610.03195/) (2026)

## Abstract (arXiv)

Large language models have advanced from single-turn question answering to deep research systems that iteratively decompose research questions, invoke retrieval tools, and synthesize information across multiple rounds. Evaluating such systems typically involves scoring their final research reports holistically, but this end-to-end paradigm tightly couples the language model's decision-making, workflow design, and environmental feedback, precluding decomposable analysis of individual components. We introduce ScholarGym, an evaluation environment that isolates the information-gathering stage of deep research on academic literature. Under a unified workflow, ScholarGym decomposes the research process into three explicit stages -- Query Planning, Tool Invocation, and Relevance Assessment -- and evaluates each against 2,536 expert-annotated queries over a static corpus of 570K papers with deterministic retrieval. Systematic experiments reveal that iterative query decomposition yields 2.9--3.3× F1 gains over single-query retrieval, models with extended thinking trade recall for precision, and Query Planning quality together with Relevance Assessment constitute dual bottlenecks that separate proprietary from open-source model performance.