mlsanityData to 14 Oct 2026

ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

arXiv 2503.21248 ↗First posted 27 Mar 2025cs.CL cs.AI cs.CEcited by 66

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses remains unexamined due to the lack of a dedicated benchmark. To address this gap, we introduce the first large-scale benchmark for evaluating LLMs on a sufficient set of scientific discovery sub-tasks-inspiration retrieval, hypothesis composition, and hypothesis ranking-where sufficient means that perfectly solving these sub-tasks perfectly solves the overall discovery task. We develop an automated LLM-based framework that extracts critical components-research questions, background surveys, inspirations, and hypotheses-from papers across 12 disciplines, with expert validation confirming its accuracy. To prevent data contamination, we focus exclusively on publications from 2024 onward, ensuring minimal overlap with LLM pretraining data; our automated framework further enables automatic extraction of even more recent papers as LLM pretraining cutoffs advance, supporting scalable and contamination-free automatic renewal of this discovery benchmark. Our evaluation shows that, across disciplines, LLMs excel at inspiration retrieval-an out-of-distribution task-suggesting their ability to surface novel knowledge associations.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 11 of its 31 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

11 of 31 references.

References · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  4. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  5. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  6. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  7. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  8. 2025LLM4SR: A Survey on Large Language Models for Scientific Researchcited by 83
  9. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  10. 2024MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypothesescited by 72
  11. 2024Can Large Language Models Unlock Novel Scientific Research Ideas?cited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 44.

66 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  3. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
  4. 2026EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discoverycited by 51
  5. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 43
  6. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 38
  7. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  8. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  9. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 22
  10. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 21
  11. 2026ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidencecited by 19
  12. 2026MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AIcited by 17
38 more
  1. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 16
  2. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  3. 2025MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Searchcited by 15
  4. 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvementscited by 15
  5. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysiscited by 14
  6. 2025DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generationcited by 12
  7. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 11
  8. 2025A Survey of AI Scientistscited by 11
  9. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  10. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  11. 2026Measuring the Gap Between Human and LLM Research Ideascited by 10
  12. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  13. 2026On the Limits of LLM-as-Judge for Scientific Novelty Assessmentcited by 7
  14. 2025MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedbackcited by 6
  15. 2025Harnessing Large Language Models for Scientific Novelty Detectioncited by 6
  16. 2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literaturecited by 6
  17. 2026CausalGame: Benchmarking Causal Thinking of LLM Agents in Gamescited by 5
  18. 2026MOOSE-Star: Unlocking Tractable Training for Scientific Discovery by Breaking the Complexity Barriercited by 4
  19. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  20. 2025Augmenting Research Ideation with Data: An Empirical Investigation in Social Sciencecited by 3
  21. 2026InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systemscited by 3
  22. 2026Scientific reasoning does not reliably translate into scientific forecasting in frontier AIcited by 3
  23. 2025ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflowscited by 2
  24. 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Taskscited by 2
  25. 2026AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classificationcited by 2
  26. 2026ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgmentcited by 2
  27. 2026ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomescited by 2
  28. 2026Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generationcited by 2
  29. 2026What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideationcited by 2
  30. 2025IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Researchcited by 1
  31. 2026Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discoverycited by 1
  32. 2026Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Sciencecited by 1
  33. 2026TCA-SIR: Learning Target-Conditioned Abstractions for Scientific Inspiration Retrievalcited by 1
  34. 2026ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchscited by 1
  35. 2026Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographiescited by 1
  36. 2026Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Rankingcited by 1
  37. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessmentcited by 1
  38. 2025Beyond Specialization: Benchmarking LLMs for Transliteration of Indian Languagescited by 0

50 most cited of 66 All 66 (JSON)