# Evaluating Large Language Models in Scientific Discovery

arXiv: https://arxiv.org/abs/2512.15567 · HTML: https://mlsanity.com/paper/2512.15567/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2025.

## Built on by (most cited first, 30 of 42)

- [Accelerating Scientific Discovery with Autonomous Goal-evolving Agents](https://mlsanity.com/paper/2512.21782/) (2025)
- [Benchmarking AI Agents for Addressing Scientific Challenges Across Scales](https://mlsanity.com/paper/2606.12736/) (2026)
- [FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs](https://mlsanity.com/paper/2512.20732/) (2025)
- [Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents](https://mlsanity.com/paper/2601.15322/) (2026)
- [MADE: Benchmark Environments for Closed-Loop Materials Discovery](https://mlsanity.com/paper/2601.20996/) (2026)
- [Principle-Evolvable Scientific Discovery via Uncertainty Minimization](https://mlsanity.com/paper/2602.06448/) (2026)
- [Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models](https://mlsanity.com/paper/2608.09696/) (2026)
- [When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs](https://mlsanity.com/paper/2602.03554/) (2026)
- [Human Cognition in Machines: A Unified Perspective of World Models](https://mlsanity.com/paper/2604.16592/) (2026)
- [Sanity Checks for Agentic Data Science](https://mlsanity.com/paper/2604.11003/) (2026)
- [Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs](https://mlsanity.com/paper/2606.32032/) (2026)
- [Towards Diverse Scientific Hypothesis Search with Large Language Models](https://mlsanity.com/paper/2606.10587/) (2026)
- [Discovery Foundation Models: Toward Open-Ended Discovery Intelligence](https://mlsanity.com/paper/2609.15973/) (2026)
- [Knowing When Not to Answer: Abstention-Aware Scientific Reasoning](https://mlsanity.com/paper/2602.14189/) (2026)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoning](https://mlsanity.com/paper/2603.03790/) (2026)
- [SciMDR: Advancing Scientific Multimodal Document Reasoning](https://mlsanity.com/paper/2603.12249/) (2026)
- [More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processes](https://mlsanity.com/paper/2603.24877/) (2026)
- [MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation](https://mlsanity.com/paper/2603.25253/) (2026)
- [Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?](https://mlsanity.com/paper/2605.28778/) (2026)
- [Quantifying Faithful Confidence Expression in Large Reasoning Models](https://mlsanity.com/paper/2606.03969/) (2026)
- [K-Bench: measuring model performance on real scientific agent requests](https://mlsanity.com/paper/2608.21601/) (2026)
- [Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Science](https://mlsanity.com/paper/2603.28361/) (2026)
- [Heterogeneous Scientific Foundation Model Collaboration](https://mlsanity.com/paper/2604.27351/) (2026)
- [Agentic-imodels: Evolving agentic interpretability tools via autoresearch](https://mlsanity.com/paper/2605.03808/) (2026)
- [Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning](https://mlsanity.com/paper/2605.07251/) (2026)
- [Metacognition in LLMs: Foundations, Progress, and Opportunities](https://mlsanity.com/paper/2607.11881/) (2026)
- [La Agente 'Optima: Towards Agentic Self-Driving Laboratories](https://mlsanity.com/paper/2609.04564/) (2026)
- [The Plausibility Trap: Using Probabilistic Engines for Deterministic Tasks](https://mlsanity.com/paper/2601.15130/) (2026)
- [Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methods](https://mlsanity.com/paper/2602.17027/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly applied to scientific research, yet prevailing science benchmarks probe decontextualized knowledge and overlook the iterative reasoning, hypothesis generation, and observation interpretation that drive scientific discovery. We introduce a scenario-grounded benchmark that evaluates LLMs across biology, chemistry, materials, and physics, where domain experts define research projects of genuine interest and decompose them into modular research scenarios from which vetted questions are sampled. The framework assesses models at two levels: (i) question-level accuracy on scenario-tied items and (ii) project-level performance, where models must propose testable hypotheses, design simulations or experiments, and interpret results. Applying this two-phase scientific discovery evaluation (SDE) framework to state-of-the-art LLMs reveals a consistent performance gap relative to general science benchmarks, diminishing return of scaling up model sizes and reasoning, and systematic weaknesses shared across top-tier models from different providers. Large performance variation in research scenarios leads to changing choices of the best performing model on scientific discovery projects evaluated, suggesting all current LLMs are distant to general scientific "superintelligence". Nevertheless, LLMs already demonstrate promise in a great variety of scientific discovery projects, including cases where constituent scenario scores are low, highlighting the role of guided exploration and serendipity in discovery. This SDE framework offers a reproducible benchmark for discovery-relevant evaluation of LLMs and charts practical paths to advance their development toward scientific discovery.