# ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

arXiv: https://arxiv.org/abs/2503.21248 · HTML: https://mlsanity.com/paper/2503.21248/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023)
- [LLM4SR: A Survey on Large Language Models for Scientific Research](https://mlsanity.com/paper/2501.04306/) (2025)
- [MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures](https://mlsanity.com/paper/2406.06565/) (2024)
- [MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses](https://mlsanity.com/paper/2410.07076/) (2024)
- [Can Large Language Models Unlock Novel Scientific Research Ideas?](https://mlsanity.com/paper/2409.06185/) (2024)

## Built on by (most cited first, 30 of 66)

- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents](https://mlsanity.com/paper/2511.07685/) (2025)
- [EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery](https://mlsanity.com/paper/2603.08127/) (2026)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents](https://mlsanity.com/paper/2602.06855/) (2026)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks](https://mlsanity.com/paper/2509.01396/) (2025)
- [ResearchGym: Evaluating Language Model Agents on Real-World AI Research](https://mlsanity.com/paper/2602.15112/) (2026)
- [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025)
- [ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence](https://mlsanity.com/paper/2605.26340/) (2026)
- [MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI](https://mlsanity.com/paper/2605.08678/) (2026)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights](https://mlsanity.com/paper/2602.02905/) (2026)
- [MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search](https://mlsanity.com/paper/2505.19209/) (2025)
- [The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements](https://mlsanity.com/paper/2506.22419/) (2025)
- [FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis](https://mlsanity.com/paper/2510.13936/) (2025)
- [DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation](https://mlsanity.com/paper/2512.17776/) (2025)
- [MIR: Methodology Inspiration Retrieval for Scientific Research Problems](https://mlsanity.com/paper/2506.00249/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery](https://mlsanity.com/paper/2605.23204/) (2026)
- [Measuring the Gap Between Human and LLM Research Ideas](https://mlsanity.com/paper/2607.01233/) (2026)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [On the Limits of LLM-as-Judge for Scientific Novelty Assessment](https://mlsanity.com/paper/2606.12071/) (2026)
- [MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback](https://mlsanity.com/paper/2505.17873/) (2025)
- [Harnessing Large Language Models for Scientific Novelty Detection](https://mlsanity.com/paper/2505.24615/) (2025)
- [WildSci: Advancing Scientific Reasoning from In-the-Wild Literature](https://mlsanity.com/paper/2601.05567/) (2026)
- [CausalGame: Benchmarking Causal Thinking of LLM Agents in Games](https://mlsanity.com/paper/2607.04293/) (2026)
- [MOOSE-Star: Unlocking Tractable Training for Scientific Discovery by Breaking the Complexity Barrier](https://mlsanity.com/paper/2603.03756/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. To address this gap, we introduce the first large-scale benchmark for evaluating LLMs on a sufficient set of scientific discovery sub-tasks-inspiration retrieval, hypothesis composition, and hypothesis ranking-where sufficient means that perfectly solving these sub-tasks perfectly solves the overall discovery task. We develop an automated LLM-based framework that extracts critical components-research questions, background surveys, inspirations, and hypotheses-from papers across 12 disciplines, with expert validation confirming its accuracy. To prevent data contamination, we focus exclusively on publications from 2024 onward, ensuring minimal overlap with LLM pretraining data; our automated framework further enables automatic extraction of even more recent papers as LLM pretraining cutoffs advance, supporting scalable and contamination-free automatic renewal of this discovery benchmark. Our evaluation shows that, across disciplines, LLMs excel at inspiration retrieval-an out-of-distribution task-suggesting their ability to surface novel knowledge associations.