# Data-driven Discovery with Large Generative Models

arXiv: https://arxiv.org/abs/2402.13610 · HTML: https://mlsanity.com/paper/2402.13610/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 117 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 30 of 49)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning](https://mlsanity.com/paper/2006.08381/) (2020)
- [Explaining Answers with Entailment Trees](https://mlsanity.com/paper/2104.08661/) (2021)
- [CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization](https://mlsanity.com/paper/2310.10134/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)

## Built on by (most cited first, 30 of 34)

- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [DiscoveryBench: Towards Data-Driven Discovery with Large Language Models](https://mlsanity.com/paper/2407.01725/) (2024)
- [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024)
- [LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models](https://mlsanity.com/paper/2504.10415/) (2025)
- [Autonomous LLM-driven research from data to human-verifiable research papers](https://mlsanity.com/paper/2404.17605/) (2024)
- [Cocoa: Co-Planning and Co-Execution with AI Agents](https://mlsanity.com/paper/2412.10999/) (2024)
- [AutoDiscovery: Open-ended Scientific Discovery via Bayesian Surprise](https://mlsanity.com/paper/2507.00310/) (2025)
- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research](https://mlsanity.com/paper/2507.13300/) (2025)
- [LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs](https://mlsanity.com/paper/2605.24043/) (2026)
- [AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists](https://mlsanity.com/paper/2506.08140/) (2025)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research](https://mlsanity.com/paper/2505.16100/) (2025)
- [Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery](https://mlsanity.com/paper/2508.17380/) (2025)
- [BIASINSPECTOR: Detecting Bias in Structured Data through LLM Agents](https://mlsanity.com/paper/2504.04855/) (2025)
- [Generating Literature-Driven Scientific Theories at Scale](https://mlsanity.com/paper/2601.16282/) (2026)
- [Augmenting Research Ideation with Data: An Empirical Investigation in Social Science](https://mlsanity.com/paper/2505.21396/) (2025)
- [Research Borderlands: Analysing Writing Across Research Cultures](https://mlsanity.com/paper/2506.00784/) (2025)
- [MaD Physics: Evaluating information seeking under constraints in physical environments](https://mlsanity.com/paper/2605.10820/) (2026)
- [Recovering Wasted Compute in Autoresearch Agents](https://mlsanity.com/paper/2608.10424/) (2026)
- [LLM Agents for Interactive Exploration of Historical Cadastre Data: Framework and Application to Venice](https://mlsanity.com/paper/2505.17148/) (2025)
- [Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing](https://mlsanity.com/paper/2511.12529/) (2025)
- [SciDER: Scientific Data-centric End-to-end Researcher](https://mlsanity.com/paper/2603.01421/) (2026)
- [Analytica: Soft Propositional Reasoning for Robust and Scalable LLM-Driven Analysis](https://mlsanity.com/paper/2604.23072/) (2026)
- [D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery](https://mlsanity.com/paper/2604.27977/) (2026)
- [Evidence-Informed LLM Beliefs for Continual Scientific Discovery](https://mlsanity.com/paper/2606.29182/) (2026)
- [On the Role of Model Prior in Real-World Inductive Reasoning](https://mlsanity.com/paper/2412.13645/) (2024)
- [CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents](https://mlsanity.com/paper/2512.01089/) (2025)
- [Discovery Agents for Real-Time Analytics: Toward Proactive Insight Systems](https://mlsanity.com/paper/2605.27571/) (2026)
- [ClinLens: Towards Long-Horizon Coding Agents for Longitudinal Multimodal Clinical Data Science](https://mlsanity.com/paper/2607.26155/) (2026)

## Abstract (arXiv)

With the accumulation of data at an unprecedented rate, its potential to fuel scientific discovery is growing exponentially. This position paper urges the Machine Learning (ML) community to exploit the capabilities of large generative models (LGMs) to develop automated systems for end-to-end data-driven discovery -- a paradigm encompassing the search and verification of hypotheses purely from a set of provided datasets, without the need for additional data collection or physical experiments. We first outline several desiderata for an ideal data-driven discovery system. Then, through DATAVOYAGER, a proof-of-concept utilizing GPT-4, we demonstrate how LGMs fulfill several of these desiderata -- a feat previously unattainable -- while also highlighting important limitations in the current system that open up opportunities for novel ML research. We contend that achieving accurate, reliable, and robust end-to-end discovery systems solely through the current capabilities of LGMs is challenging. We instead advocate for fail-proof tool integration, along with active user moderation through feedback mechanisms, to foster data-driven scientific discoveries with efficiency and reproducibility.