# Can AI Agents Synthesize Scientific Conclusions?

arXiv: https://arxiv.org/abs/2606.11337 · HTML: https://mlsanity.com/paper/2606.11337/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 133 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 30 of 33)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [WebThinker: Empowering Large Reasoning Models with Deep Research Capability](https://mlsanity.com/paper/2504.21776/) (2025)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025)
- [Benchmarking Benchmark Leakage in Large Language Models](https://mlsanity.com/paper/2404.18824/) (2024)
- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents](https://mlsanity.com/paper/2509.06501/) (2025)
- [OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs](https://mlsanity.com/paper/2411.14199/) (2024)
- [The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas](https://mlsanity.com/paper/2506.20803/) (2025)
- [WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research](https://mlsanity.com/paper/2509.13312/) (2025)
- [The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer](https://mlsanity.com/paper/2502.15631/) (2025)
- [ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks](https://mlsanity.com/paper/2508.15804/) (2025)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry](https://mlsanity.com/paper/2507.16280/) (2025)
- [ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists](https://mlsanity.com/paper/2506.01241/) (2025)
- [The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality](https://mlsanity.com/paper/2512.10791/) (2025)
- [Exploring ChatGPT for Toxicity Detection in GitHub](https://mlsanity.com/paper/2312.13105/) (2023)
- [The Adoption and Usage of AI Agents: Early Evidence from Perplexity](https://mlsanity.com/paper/2512.07828/) (2025)
- [Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios](https://mlsanity.com/paper/2401.05761/) (2024)
- [Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?](https://mlsanity.com/paper/2506.18322/) (2025)
- [Can Large Language Models Match the Conclusions of Systematic Reviews?](https://mlsanity.com/paper/2505.22787/) (2025)
- [Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines](https://mlsanity.com/paper/2512.09483/) (2025)
- [ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios](https://mlsanity.com/paper/2510.10998/) (2025)

## Built on by (most cited first, 1 of 1)

- [BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies](https://mlsanity.com/paper/2610.07614/) (2026)

## Abstract (arXiv)

Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.