# DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality

arXiv: https://arxiv.org/abs/2603.05912 · HTML: https://mlsanity.com/paper/2603.05912/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 35)

- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- ["Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection](https://mlsanity.com/paper/1705.00648/) (2017)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents](https://mlsanity.com/paper/2506.11763/) (2025)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs](https://mlsanity.com/paper/2311.17371/) (2023)
- [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025)
- [Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers](https://mlsanity.com/paper/2311.09000/) (2023)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [AmbiFC: Fact-Checking Ambiguous Claims with Evidence](https://mlsanity.com/paper/2104.00640/) (2021)
- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [Characterizing Deep Research: A Benchmark and Formal Definition](https://mlsanity.com/paper/2508.04183/) (2025)
- [ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists](https://mlsanity.com/paper/2506.01241/) (2025)
- [Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance](https://mlsanity.com/paper/2410.18889/) (2024)
- [CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators](https://mlsanity.com/paper/2210.06812/) (2022)

## Built on by (most cited first, 10 of 10)

- [MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome](https://mlsanity.com/paper/2603.28407/) (2026)
- [Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI](https://mlsanity.com/paper/2607.22368/) (2026)
- [Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions](https://mlsanity.com/paper/2607.20891/) (2026)
- [ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research](https://mlsanity.com/paper/2605.28282/) (2026)
- [Self-Modifying Lean Proof Agents with Verifier-Grounded Benchmark Coevolution](https://mlsanity.com/paper/2607.17352/) (2026)
- [Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation](https://mlsanity.com/paper/2608.06718/) (2026)
- [An Interactive Agent for Requirement-Driven Candidate Sourcing](https://mlsanity.com/paper/2608.23501/) (2026)
- [Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning](https://mlsanity.com/paper/2609.06027/) (2026)
- [Lazy Grounding: Attacking Search Agents with Factual Evidence](https://mlsanity.com/paper/2608.30303/) (2026)
- [Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents](https://mlsanity.com/paper/2609.39026/) (2026)

## Abstract (arXiv)

Search-augmented LLM agents can produce deep research reports (DRRs), but verifying claim-level factuality remains challenging. Existing fact-checkers are primarily designed for general-domain, factoid-style atomic claims, and there is no benchmark to test whether such verifiers transfer to DRRs. Yet building such a benchmark is itself difficult. We first show that static expert-labeled benchmarks are brittle in this setting: in a controlled study with PhD-level specialists, unassisted experts achieve only 60.8% accuracy on a hidden micro-gold set of verifiable claims. We propose Evolving Benchmarking via Audit-then-Score (AtS), where benchmark labels and rationales are explicitly revisable: when a verifier disagrees with the current benchmark, it must submit evidence; an auditor adjudicates the dispute; and accepted revisions update the benchmark before models are scored. Across four AtS rounds, expert micro-gold accuracy rises to 90.9%, indicating experts are substantially more reliable as auditors than as one-shot labelers. We instantiate AtS as DeepFact-Bench, a versioned DRR factuality benchmark with auditable rationales, and DeepFact-Eval, a document-level verification agent (with a grouped lite variant) that outperforms existing verifiers on DeepFact-Bench and transfers well to external factuality datasets.