# LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models

arXiv: https://arxiv.org/abs/2504.10415 · HTML: https://mlsanity.com/paper/2504.10415/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients](https://mlsanity.com/paper/1912.04871/) (2019)
- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Neural Symbolic Regression that Scales](https://mlsanity.com/paper/2106.06427/) (2021)
- [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023)
- [On Memorization of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2410.23123/) (2024)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery](https://mlsanity.com/paper/2405.09783/) (2024)
- [Symbolic Regression with a Learned Concept Library](https://mlsanity.com/paper/2409.09359/) (2024)
- [Transformer-based Planning for Symbolic Regression](https://mlsanity.com/paper/2303.06833/) (2023)
- [Symbolic Regression is NP-hard](https://mlsanity.com/paper/2207.01018/) (2022)
- [Symbolic Physics Learner: Discovering governing equations via Monte Carlo tree search](https://mlsanity.com/paper/2205.13134/) (2022)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges](https://mlsanity.com/paper/2412.11427/) (2024)
- [Data-driven Discovery with Large Generative Models](https://mlsanity.com/paper/2402.13610/) (2024)
- [In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery](https://mlsanity.com/paper/2404.19094/) (2024)
- [AutoTurb: Using Large Language Models for Automatic Algebraic Model Discovery of Turbulence Closure](https://mlsanity.com/paper/2410.10657/) (2024)

## Built on by (most cited first, 30 of 78)

- [AI4Research: A Survey of Artificial Intelligence for Scientific Research](https://mlsanity.com/paper/2507.01903/) (2025)
- [A Survey on Large Language Model Benchmarks](https://mlsanity.com/paper/2508.15361/) (2025)
- [NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents](https://mlsanity.com/paper/2510.07172/) (2025)
- [SR-Scientist: Scientific Equation Discovery With Agentic AI](https://mlsanity.com/paper/2510.11661/) (2025)
- [Deep Research: A Systematic Survey](https://mlsanity.com/paper/2512.02038/) (2025)
- [Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows](https://mlsanity.com/paper/2512.16969/) (2025)
- [DeltaEvolve: Accelerating Scientific Discovery through Momentum-Driven Evolution](https://mlsanity.com/paper/2602.02919/) (2026)
- [Paper2Video: Automatic Video Generation from Scientific Papers](https://mlsanity.com/paper/2510.05096/) (2025)
- [DrSR: LLM based Scientific Equation Discovery with Dual Reasoning from Data and Experience](https://mlsanity.com/paper/2506.04282/) (2025)
- [Structured Scaling of AI Discovery Across Diverse Scientific Domains](https://mlsanity.com/paper/2604.19341/) (2026)
- [Think like a Scientist: Physics-guided LLM Agent for Equation Discovery](https://mlsanity.com/paper/2602.12259/) (2026)
- [PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors](https://mlsanity.com/paper/2507.15550/) (2025)
- [Can Language Models Discover Scaling Laws?](https://mlsanity.com/paper/2507.21184/) (2025)
- [LLEMA: Evolutionary Search with LLMs for Multi-Objective Materials Discovery](https://mlsanity.com/paper/2510.22503/) (2025)
- [AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists](https://mlsanity.com/paper/2506.08140/) (2025)
- [Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab](https://mlsanity.com/paper/2507.02083/) (2025)
- [LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs](https://mlsanity.com/paper/2605.24043/) (2026)
- [SUPERChem: A Multimodal Reasoning Benchmark in Chemistry](https://mlsanity.com/paper/2512.01274/) (2025)
- [DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking](https://mlsanity.com/paper/2605.26087/) (2026)
- [FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs](https://mlsanity.com/paper/2512.20732/) (2025)
- [Discovering Symbolic Differential Equations with Symmetry Invariants](https://mlsanity.com/paper/2505.12083/) (2025)
- [Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery](https://mlsanity.com/paper/2508.17380/) (2025)
- [The Need for Verification in AI-Driven Scientific Discovery](https://mlsanity.com/paper/2509.01398/) (2025)
- [HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Bounds](https://mlsanity.com/paper/2510.15614/) (2025)
- [LLM-ACES: Closed-Loop Discovery of Dynamical Systems with LLM-Guided Adaptive Search](https://mlsanity.com/paper/2606.25039/) (2026)
- [Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks](https://mlsanity.com/paper/2606.29082/) (2026)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [Position: Biology is the Challenge Physics-Informed ML Needs to Evolve](https://mlsanity.com/paper/2510.25368/) (2025)
- [BenchEvolver: Frontier Task Synthesis via Solution-Centric Evolution](https://mlsanity.com/paper/2606.01286/) (2026)
- [Analyzing Generalization in Pre-Trained Symbolic Regression](https://mlsanity.com/paper/2509.19849/) (2025)

## Abstract (arXiv)

Scientific equation discovery is a fundamental task in the history of scientific progress, enabling the derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their potential to leverage embedded scientific knowledge for hypothesis generation. However, evaluating the true discovery capabilities of these methods remains challenging, as existing benchmarks often rely on common equations that are susceptible to memorization by LLMs, leading to inflated performance metrics that do not reflect discovery. In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorized forms, and LSR-Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning. Through extensive evaluation of several state-of-the-art methods, using both open and closed LLMs, we find that the best-performing system so far achieves only 31.5% symbolic accuracy. These findings highlight the challenges of scientific equation discovery, positioning LLM-SRBench as a valuable resource for future research.