mlsanityData to 14 Oct 2026

DiscoveryBench: Towards Data-Driven Discovery with Large Language Models

arXiv 2407.01725 ↗First posted 1 Jul 2024cs.CL cs.AI cs.LGcited by 109

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets? To evaluate this question, we present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. The benchmark is designed to systematically assess current model capabilities in discovery tasks and provide a useful resource for improving them. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering, by manually deriving discovery workflows from published papers to approximate the real-world challenges faced by researchers, where each task is defined by a dataset, its metadata, and a discovery goal in natural language. We additionally provide 903 synthetic tasks to conduct controlled evaluations across task complexity. Furthermore, our structured formalism of data-driven discovery enables a facet-based evaluation that provides useful insights into different failure modes. We evaluate several popular LLM-based reasoning frameworks using both open and closed LLMs as baselines on DiscoveryBench and find that even the best system scores only 25%. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 67 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

22 of 67 references.

References · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2023GPTScore: Evaluate as You Desirecited by 361
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  10. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  11. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 117
  12. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 95
10 more
  1. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  2. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  3. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 63
  4. 2022AMLB: an AutoML Benchmarkcited by 55
  5. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 50
  6. 2024Automated Statistical Model Discovery with Language Modelscited by 41
  7. 2024Data-driven Discovery with Large Generative Modelscited by 34
  8. 2024Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verificationcited by 13
  9. 2023Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Modelscited by 11
  10. 2023Status Quo and Problems of Requirements Engineering for Machine Learning: Results from an International Surveycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 34; 2026: 71.

109 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  2. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  3. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  4. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  5. 2025BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biologycited by 89
  6. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  7. 2025LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Modelscited by 71
  8. 2026Code as Agent Harnesscited by 47
  9. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  10. 2025R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Sciencecited by 41
  11. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  12. 2025AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprisecited by 33
38 more
  1. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 28
  2. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  3. 2024Aviary: training language agents on challenging scientific taskscited by 26
  4. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 25
  5. 2025Exploring LLM Agents for Cleaning Tabular Machine Learning Datasetscited by 20
  6. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  7. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  8. 2025CellForge: Agentic Design of Virtual Cell Modelscited by 13
  9. 2026HeurekaBench: A Benchmarking Framework for AI Co-scientistcited by 12
  10. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  11. 2025Scaling Generalist Data-Analytic Agentscited by 11
  12. 2025A Survey of AI Scientistscited by 11
  13. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  14. 2026How Well Does Agent Development Reflect Real-World Work?cited by 10
  15. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  16. 2026DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinkingcited by 10
  17. 2025Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluatorcited by 9
  18. 2025DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discoverycited by 9
  19. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  20. 2026Training AI Scientists to Replicate Researchcited by 9
  21. 2025BioDSA-1K: Benchmarking Data Science Agents for Biomedical Researchcited by 8
  22. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Researchcited by 8
  23. 2026How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Taskscited by 8
  24. 2025Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Studycited by 7
  25. 2025Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discoverycited by 7
  26. 2025JoyAgent-JDGenie: Technical Report on the GAIAcited by 7
  27. 2026CREATE: Testing LLMs for Associative Creativitycited by 7
  28. 2026DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytellingcited by 7
  29. 2025K-Dense Analyst: Towards Fully Automated Scientific Analysiscited by 6
  30. 2026ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecuritycited by 6
  31. 2025BioDisco: Multi-agent hypothesis generation with dual-mode evidence, iterative feedback and temporal evaluationcited by 5
  32. 2025CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamicscited by 5
  33. 2025BIASINSPECTOR: Detecting Bias in Structured Data through LLM Agentscited by 4
  34. 2025Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspectivecited by 4
  35. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  36. 2025PCS Workflow for Veridical Data Science in the Age of AIcited by 4
  37. 2026ASI-Bench: At the Dawn of Artificial Superintelligencecited by 4
  38. 2026ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environmentscited by 4

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
  2. 2024OCDB: Revisiting Causal Discovery with a Comprehensive Benchmark and Evaluation Framework
  3. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Models
  4. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  5. 2026FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
  6. 2025Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures
  7. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science
  8. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination