Structural Enforcement of Statistical Rigor in AI-Driven Discovery: A Functional Architecture
Not on the map. Why?
Abstract (arXiv)
AI-Scientist systems risk manufacturing spurious discoveries through uncontrolled multiple testing. We present a functional architecture that enforces statistical rigor at two levels: a Haskell embedded domain-specific language (the Research monad) that makes it impossible to test a hypothesis without updating the error budget, and a declarative scaffold that fixes the data flow and the statistical test, together with an OS-level sandbox that makes validation data physically absent from the environment in which LLM-generated code runs. We treat FDR control as a formal requirement and trace it to the implementation. We ground the design in a machine-checked Lean~4 formalization of LORD online false-discovery-rate (FDR) control: we derive its error budget and prove marginal FDR control, and full FDR control when thresholds do not adapt to earlier rejections. We then verify in SPARK/Ada that the LORD thresholds, computed in IEEE~754 arithmetic, never exceed the available wealth, given a margin condition that our configurations meet by a factor of at least eight; without a margin the property fails. To our knowledge this is the first machine-checked proof of an online FDR control theorem. In simulation, the architecture holds the false discovery rate near 1% against a 5% target, where a naive approach reaches 41%. In end-to-end case studies, a valid test avoids the false discoveries a flawed one produces, yet still finds real effects when the data allow. An adversarial evaluation confirms that, inside the sandbox, generated code cannot read the held-out data even when given its exact path.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 56 references that are in the map. First posted on arXiv 10 Nov 2025.
Builds on
19 of 56 references.
References · 19
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
- 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
- 2025The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Searchcited by 422
- 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
- 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 77
- 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
- 2025Deep Research: A Survey of Autonomous Research Agentscited by 66
- 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 63
- 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 33
- 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
- 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
- 2025ToolUniverse: An open platform for democratizing AI scientistscited by 24
7 more
- 2016Online Rules for Control of False Discovery Rate and False Discovery Exceedancecited by 21
- 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 16
- 2017Online control of the false discovery rate with decaying memorycited by 11
- 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
- 2018Asynchronous Online Testing of Multiple Hypothesescited by 10
- 2025Scientific Hypothesis Generation and Validation: Methods, Datasets, and Future Directionscited by 9
- 2019Tea: A High-level Language and Runtime System for Automating Statistical Analysiscited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.