# HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation

arXiv: https://arxiv.org/abs/2504.11524 · HTML: https://mlsanity.com/paper/2504.11524/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts](https://mlsanity.com/paper/2205.01833/) (2022)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents](https://mlsanity.com/paper/2406.06769/) (2024)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024)
- [Goal Driven Discovery of Distributional Differences via Language Descriptions](https://mlsanity.com/paper/2302.14233/) (2023)
- [Describing Differences in Image Sets with Natural Language](https://mlsanity.com/paper/2312.02974/) (2023)
- [LitSearch: A Retrieval Benchmark for Scientific Literature Search](https://mlsanity.com/paper/2407.18940/) (2024)
- [CiteME: Can Language Models Accurately Cite Scientific Claims?](https://mlsanity.com/paper/2407.12861/) (2024)
- [Sparse Autoencoders for Hypothesis Generation](https://mlsanity.com/paper/2502.04382/) (2025)
- [Explaining Datasets in Words: Statistical Models with Natural Language Parameters](https://mlsanity.com/paper/2409.08466/) (2024)
- [Literature Meets Data: A Synergistic Approach to Hypothesis Generation](https://mlsanity.com/paper/2410.17309/) (2024)
- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [Concept Bottleneck Models Without Predefined Concepts](https://mlsanity.com/paper/2407.03921/) (2024)
- [InductionBench: LLMs Fail in the Simplest Complexity Class](https://mlsanity.com/paper/2502.15823/) (2025)
- [Google Scholar is manipulatable](https://mlsanity.com/paper/2402.04607/) (2024)

## Built on by (most cited first, 30 of 37)

- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [ResearchGym: Evaluating Language Model Agents on Real-World AI Research](https://mlsanity.com/paper/2602.15112/) (2026)
- [SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks](https://mlsanity.com/paper/2507.01001/) (2025)
- [Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics](https://mlsanity.com/paper/2510.09901/) (2025)
- [Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator](https://mlsanity.com/paper/2507.11810/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments](https://mlsanity.com/paper/2601.07606/) (2026)
- [CREATE: Testing LLMs for Associative Creativity](https://mlsanity.com/paper/2603.09970/) (2026)
- [HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds](https://mlsanity.com/paper/2510.15614/) (2025)
- [The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research](https://mlsanity.com/paper/2602.18458/) (2026)
- [BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluation](https://mlsanity.com/paper/2508.01285/) (2025)
- [Generating Literature-Driven Scientific Theories at Scale](https://mlsanity.com/paper/2601.16282/) (2026)
- [CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists](https://mlsanity.com/paper/2605.26029/) (2026)
- [ASI-Bench: At the Dawn of Artificial Superintelligence](https://mlsanity.com/paper/2608.17271/) (2026)
- [MATRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science](https://mlsanity.com/paper/2602.00376/) (2026)
- [Accelerating Social Science Research via Agentic Hypothesization and Experimentation](https://mlsanity.com/paper/2602.07983/) (2026)
- [Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation](https://mlsanity.com/paper/2604.28031/) (2026)
- [The Measurement Revolution? Credible Measurement and Inference in the Age of AI](https://mlsanity.com/paper/2608.23524/) (2026)
- [A2P-Vis: an Analyzer-to-Presenter Agentic Pipeline for Visual Insights Generation and Reporting](https://mlsanity.com/paper/2512.22101/) (2025)
- [Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation](https://mlsanity.com/paper/2608.01666/) (2026)
- [What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation](https://mlsanity.com/paper/2608.22948/) (2026)
- [HypoForge: A Self-Improving Multi-Agent Framework for Automated Hypothesis Generation and Testing via Scientific Skill Learning](https://mlsanity.com/paper/2608.25770/) (2026)
- [Can Large Language Models Forecast What Researchers Study Next?](https://mlsanity.com/paper/2609.00747/) (2026)
- [Language Models Should be Used to Surface the Unwritten Code of Science and Society](https://mlsanity.com/paper/2505.18942/) (2025)
- [Interestingness First Classifiers](https://mlsanity.com/paper/2508.19780/) (2025)
- [Scaling Unverifiable Rewards: A Case Study on Visual Insights](https://mlsanity.com/paper/2512.22650/) (2025)
- [Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery](https://mlsanity.com/paper/2607.15766/) (2026)
- [Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2609.14896/) (2026)
- [RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models](https://mlsanity.com/paper/2510.19698/) (2025)
- [Universe of Thoughts: Enabling Creative Reasoning with Large Language Models](https://mlsanity.com/paper/2511.20471/) (2025)

## Abstract (arXiv)

There is growing interest in hypothesis generation with large language models (LLMs). However, fundamental questions remain: what makes a good hypothesis, and how can we systematically evaluate methods for hypothesis generation? To address this, we introduce HypoBench, a novel benchmark designed to evaluate LLMs and hypothesis generation methods across multiple aspects, including practical utility, generalizability, and hypothesis discovery rate. HypoBench includes 7 real-world tasks and 5 synthetic tasks with 194 distinct datasets. We evaluate four state-of-the-art LLMs combined with six existing hypothesis-generation methods. Overall, our results suggest that existing methods are capable of discovering valid and novel patterns in the data. However, the results from synthetic datasets indicate that there is still significant room for improvement, as current hypothesis generation methods do not fully uncover all relevant or meaningful patterns. Specifically, in synthetic settings, as task difficulty increases, performance significantly drops, with best models and methods only recovering 38.8% of the ground-truth hypotheses. These findings highlight challenges in hypothesis generation and demonstrate that HypoBench serves as a valuable resource for improving AI systems designed to assist scientific discovery.