# DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research

arXiv: https://arxiv.org/abs/2505.19253 · HTML: https://mlsanity.com/paper/2505.19253/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- First posted on arXiv 25 May 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Built on by (most cited first, 30 of 53)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks](https://mlsanity.com/paper/2604.20087/) (2026)
- [WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research](https://mlsanity.com/paper/2509.13312/) (2025)
- [Characterizing Deep Research: A Benchmark and Formal Definition](https://mlsanity.com/paper/2508.04183/) (2025)
- [ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking](https://mlsanity.com/paper/2601.06487/) (2026)
- [Towards Personalized Deep Research: Benchmarks and Evaluations](https://mlsanity.com/paper/2509.25106/) (2025)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [How Far Are We from Genuinely Useful Deep Research Agents?](https://mlsanity.com/paper/2512.01948/) (2025)
- [O-Researcher: An Open Ended Deep Research Model via Multi-Agent Distillation and Agentic RL](https://mlsanity.com/paper/2601.03743/) (2026)
- [VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents](https://mlsanity.com/paper/2601.16973/) (2026)
- [Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?](https://mlsanity.com/paper/2605.19196/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents](https://mlsanity.com/paper/2507.05495/) (2025)
- [RAVine: Reality-Aligned Evaluation for Agentic Search](https://mlsanity.com/paper/2507.16725/) (2025)
- [MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome](https://mlsanity.com/paper/2603.28407/) (2026)
- [Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories](https://mlsanity.com/paper/2606.02060/) (2026)
- [Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization](https://mlsanity.com/paper/2602.22675/) (2026)
- [Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarks](https://mlsanity.com/paper/2603.06942/) (2026)
- [InfoSeeker: A Scalable Hierarchical Parallel Agent Framework for Web Information Seeking](https://mlsanity.com/paper/2604.02971/) (2026)
- [Less LLM, More Documents: Searching for Improved RAG](https://mlsanity.com/paper/2510.02657/) (2025)
- [Efficient Tree-Structured Deep Research with Adaptive Resource Allocation](https://mlsanity.com/paper/2510.05145/) (2025)
- [AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web](https://mlsanity.com/paper/2604.10938/) (2026)
- [Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions](https://mlsanity.com/paper/2607.20891/) (2026)
- [Diagnosing and Repairing Citation Failures in Generative Engine Optimization](https://mlsanity.com/paper/2603.09296/) (2026)
- [R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoning](https://mlsanity.com/paper/2604.03004/) (2026)
- [TVIR: Building Deep Research Agents Towards Text-Visual Interleaved Report Generation](https://mlsanity.com/paper/2606.02320/) (2026)
- [Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training](https://mlsanity.com/paper/2606.10709/) (2026)
- [A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis](https://mlsanity.com/paper/2506.14345/) (2025)

## Abstract (arXiv)

Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce textscDeepResearchGym as an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is free for research use. To evaluate deep research systems' outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge to measure alignment with users' information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with~textscDeepResearchGym achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A case study on short-answer search agents further demonstrates the sandbox's utility for cost-effective training, showing that models trained within the sandbox can generalize to commercial search.