mlsanityData to 14 Oct 2026

Sanity Checks for Agentic Data Science

arXiv 2604.11003 ↗First posted 13 Apr 2026cs.AI cs.LGcited by 4

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 75 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

34 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse21 Feb
  2. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agents22 Jan
  3. 2025AI-Researcher: Autonomous Scientific Innovation24 May
  4. 2025Agent Laboratory: Using LLM Agents as Research Assistants8 Jan
  5. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery7 Oct
  6. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep
  7. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Science19 Aug
  8. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning27 Feb

Other references · 26

  1. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  2. 2025Accelerating scientific discovery with Co-Scientistcited by 401
  3. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 373
  4. 2023Conformal Language Modelingcited by 170
  5. 2023Large Language Models for Automated Open-domain Scientific Hypotheses Discoverycited by 100
  6. 2025Kosmos: An AI Scientist for Autonomous Discoverycited by 92
  7. 2019Veridical Data Sciencecited by 66
  8. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 62
  9. 2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 50
  10. 2017Iterative Random Forests to detect predictive and stable high-order interactionscited by 42
  11. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 42
  12. 2025Evaluating Large Language Models in Scientific Discoverycited by 42
14 more
  1. 2025The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systemscited by 29
  2. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  3. 2020A Selective Review of Negative Control Methods in Epidemiologycited by 16
  4. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysiscited by 11
  5. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11
  6. 2025Evaluating Large Language Models as Expert Annotatorscited by 9
  7. 2026Human-AI Co-design for Clinical Prediction Modelscited by 9
  8. 2025Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Modelscited by 6
  9. 2020Boba: Authoring and Visualizing Multiverse Analysescited by 5
  10. 2024Generative causal testing to bridge data-driven models and scientific theories in language neurosciencecited by 5
  11. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agentscited by 5
  12. 2025PCS Workflow for Veridical Data Science in the Age of AIcited by 4
  13. 2025Beyond Quantification: Navigating Uncertainty in Professional AI Systemscited by 4
  14. 2026More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysiscited by 4
  2. 2026Unsupervised Skill Discovery for Agentic Data Analysiscited by 2
  3. 2026ForkSCOPE: Charting the Agentic Garden of Forking Pathscited by 0
  4. 2026Towards Reliable AI Data Scientists: Data Agents with Workflow Harnessescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic-imodels: Evolving agentic interpretability tools via autoresearch
  2. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
  3. 2026The Silent Failures in Agentic Data Science
  4. 2026Skill-based Agentic Evaluation for Real-time Data Science Tasks
  5. 2026Can AI Agents Synthesize Scientific Conclusions?
  6. 2026VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
  7. 2026Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
  8. 2026Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration