Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies
On the map: LLM agents and tool use → Web, search, and research agents
Abstract (arXiv)
Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize those papers into paper-grounded taxonomies. We introduce TaxoBench, a benchmark built from 72 highly cited LLM surveys, 3,815 cited papers, and their expert-authored taxonomies. TaxoBench evaluates systems in two settings: Deep Research mode measures end-to-end retrieval and organization from a topic, while Bottom-Up mode provides the expert paper set and isolates organization. We evaluate leaf-level assignments with ARI and V-Measure and hierarchy-level structure with US-TED, US-NTED, and Sem-Path. Across 7 Deep Research Agents and 16 LLM configurations, the best agent retrieves only 20.92% of expert-cited papers, and none of 70 standard Bottom-Up runs reaches the experts' average depth of 4.86. A controlled probe shows that models which match this depth do so by fragmenting the taxonomy, reducing alignment with the expert reference. We further find that raw Sem-Path remains near a no-organization floor even when a newer model generation gains 3.68 pp ARI; after depth matching, humans lead on all 10 matched surveys by 13.27 pp. These results identify retrieval and hierarchical organization as separate bottlenecks and show why hierarchy metrics must be calibrated before they are used to compare models.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 39 references that are in the map. First posted on arXiv 18 Jan 2026.
Builds on
26 of 39 references.
Closest ancestors · 8
Same direction, newest first
- 2026OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment4 Jan
- 2025How Far Are We from Genuinely Useful Deep Research Agents?1 Dec
- 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents10 Nov
- 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks14 Aug
- 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents23 Jun
- 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents13 Jun
- 2025Agent Laboratory: Using LLM Agents as Research Assistants8 Jan
- 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
Other references · 18
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
- 2020Fact or Fiction: Verifying Scientific Claimscited by 546
- 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
- 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
- 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
- 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
- 2025Competitive Programming with Large Reasoning Modelscited by 115
- 2024Jasper and Stella: distillation of SOTA embedding modelscited by 109
- 2025SurveyForge: On the Outline Heuristics, Memory-Driven Generation, and Multi-dimensional Evaluation for Automated Survey Writingcited by 47
- 2025MedBrowseComp: Benchmarking Medical Deep Research and Computer Usecited by 28
- 2025Deep Research: A Systematic Surveycited by 23
6 more
- 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
- 2024Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and Approachescited by 15
- 2024Taxonomy Tree Generation from Citation Graphcited by 11
- 2025STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliabilitycited by 5
- 2025SurveyBench: Can LLM(-Agents) Write Academic Surveys that Align with Reader Needs?cited by 4
- 2025TaxoAlign: Scholarly Taxonomy Generation Using Language Modelscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Deep Academic Survey: Stateful Agentic Closed-Loop Paradigm for Academic Survey Automation
- 2026SAGE: Benchmarking and Improving Retrieval for Deep Research Agents
- 2026DeepSurvey: Agent-Oriented Automated Survey Generation with Analytical Depth and Citation Reliability
- 2026STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generation
- 2025Agentic AutoSurvey: Let LLMs Survey LLMs
- 2025Deep Research Agents: A Systematic Examination And Roadmap
- 2025SurveyG: A Multi-Agent LLM Framework with Hierarchical Citation Graph for Automated Survey Generation
- 2026DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing