# Automated Hypothesis Validation with Agentic Sequential Falsifications

arXiv: https://arxiv.org/abs/2502.09858 · HTML: https://mlsanity.com/paper/2502.09858/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 69 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 30)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [MARG: Multi-Agent Review Generation for Scientific Papers](https://mlsanity.com/paper/2401.04259/) (2024)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners](https://mlsanity.com/paper/2305.14825/) (2023)
- [Large Language Models Are Not Strong Abstract Reasoners](https://mlsanity.com/paper/2305.19555/) (2023)
- [Autonomous LLM-driven research from data to human-verifiable research papers](https://mlsanity.com/paper/2404.17605/) (2024)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://mlsanity.com/paper/2407.12861/) (2024)
- [Language Models as Inductive Reasoners](https://mlsanity.com/paper/2212.10923/) (2022)
- [Inductive reasoning in humans and large language models](https://mlsanity.com/paper/2306.06548/) (2023)
- [ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor](https://mlsanity.com/paper/2406.14765/) (2024)
- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [CriticAL: Critic Automation with Language Models](https://mlsanity.com/paper/2411.06590/) (2024)

## Built on by (most cited first, 30 of 44)

- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [Robin: A multi-agent system for automating scientific discovery](https://mlsanity.com/paper/2505.13400/) (2025)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing](https://mlsanity.com/paper/2512.03109/) (2025)
- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements](https://mlsanity.com/paper/2506.22419/) (2025)
- [CellForge: Agentic Design of Virtual Cell Models](https://mlsanity.com/paper/2508.02276/) (2025)
- [HeurekaBench: A Benchmarking Framework for AI Co-scientist](https://mlsanity.com/paper/2601.01678/) (2026)
- [AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation](https://mlsanity.com/paper/2604.18240/) (2026)
- [Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics](https://mlsanity.com/paper/2510.09901/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research](https://mlsanity.com/paper/2505.16100/) (2025)
- [Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse](https://mlsanity.com/paper/2602.18710/) (2026)
- [Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models](https://mlsanity.com/paper/2608.09696/) (2026)
- [BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation](https://mlsanity.com/paper/2508.01285/) (2025)
- [AutoSci: A Memory-Centric Agentic System for the Full Scientific Research Lifecycle](https://mlsanity.com/paper/2605.31468/) (2026)
- [CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research](https://mlsanity.com/paper/2510.11985/) (2025)
- [Accelerating Social Science Research via Agentic Hypothesization and Experimentation](https://mlsanity.com/paper/2602.07983/) (2026)
- [CausalEvolve: Towards Open-Ended Discovery with Causal Scratchpad](https://mlsanity.com/paper/2603.14575/) (2026)
- [Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents](https://mlsanity.com/paper/2607.09195/) (2026)
- [Hypothesis Hunting with Evolving Networks of Autonomous Scientific Agents](https://mlsanity.com/paper/2510.08619/) (2025)
- [Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models](https://mlsanity.com/paper/2606.16999/) (2026)
- [Autonomous Scientific Discovery via Iterative Meta-Reflection](https://mlsanity.com/paper/2607.01131/) (2026)
- [Beyond Solution-Centric Search: Adaptive Inquiry and Knowledge Revision for Autonomous ML Engineering](https://mlsanity.com/paper/2608.02143/) (2026)
- [HypoForge: A Self-Improving Multi-Agent Framework for Automated Hypothesis Generation and Testing via Scientific Skill Learning](https://mlsanity.com/paper/2608.25770/) (2026)
- [EvoSCM: Scientific Belief Revision Through Causal Model Evolution and Experimentation](https://mlsanity.com/paper/2609.01526/) (2026)
- [HypoEvolve: Genetic Algorithms Enable Multi-Agent LLMs to Discover Scientific Hypotheses](https://mlsanity.com/paper/2609.15938/) (2026)
- [Automating Exploratory Multiomics Research via Language Models](https://mlsanity.com/paper/2506.07591/) (2025)
- [Why can't Epidemiology be automated (yet)?](https://mlsanity.com/paper/2507.15617/) (2025)

## Abstract (arXiv)

Hypotheses are central to information acquisition, decision-making, and discovery. However, many real-world hypotheses are abstract, high-level statements that are difficult to validate directly. This challenge is further intensified by the rise of hypothesis generation from Large Language Models (LLMs), which are prone to hallucination and produce hypotheses in volumes that make manual validation impractical. Here we propose Popper, an agentic framework for rigorous automated validation of free-form hypotheses. Guided by Karl Popper's principle of falsification, Popper validates a hypothesis using LLM agents that design and execute falsification experiments targeting its measurable implications. A novel sequential testing framework ensures strict Type-I error control while actively gathering evidence from diverse observations, whether drawn from existing data or newly conducted procedures. We demonstrate Popper on six domains including biology, economics, and sociology. Popper delivers robust error control, high power, and scalability. Furthermore, compared to human scientists, Popper achieved comparable performance in validating complex biological hypotheses while reducing time by 10 folds, providing a scalable, rigorous solution for hypothesis validation.