# Veridical Data Science

arXiv: https://arxiv.org/abs/1901.08152 · HTML: https://mlsanity.com/paper/1901.08152/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 30 of 66)

- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance](https://mlsanity.com/paper/2309.13775/) (2023)
- [Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines](https://mlsanity.com/paper/2206.03944/) (2022)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care](https://mlsanity.com/paper/2208.07406/) (2022)
- [Derandomized Novelty Detection with FDR Control via Conformal E-values](https://mlsanity.com/paper/2302.07294/) (2023)
- [Mechanistic Interpretability as Statistical Estimation: A Variance Analysis](https://mlsanity.com/paper/2510.00845/) (2025)
- [Fast Interpretable Greedy-Tree Sums](https://mlsanity.com/paper/2201.11931/) (2022)
- [Semi-Supervised Risk Control via Prediction-Powered Inference](https://mlsanity.com/paper/2412.11174/) (2024)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [The s-value: evaluating stability with respect to distributional shifts](https://mlsanity.com/paper/2105.03067/) (2021)
- [Why Do Probabilistic Clinical Models Fail To Transport Between Sites?](https://mlsanity.com/paper/2311.04787/) (2023)
- [PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework](https://mlsanity.com/paper/2505.08784/) (2025)
- [Toward a Taxonomy of Trust for Probabilistic Machine Learning](https://mlsanity.com/paper/2112.03270/) (2021)
- [Generalized Data Thinning Using Sufficient Statistics](https://mlsanity.com/paper/2303.12931/) (2023)
- [Building a stable classifier with the inflated argmax](https://mlsanity.com/paper/2405.14064/) (2024)
- [Interpretable Machine Learning for Discovery: Statistical Challenges & Opportunities](https://mlsanity.com/paper/2308.01475/) (2023)
- [Statistical Significance of Feature Importance Rankings](https://mlsanity.com/paper/2401.15800/) (2024)
- [Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse](https://mlsanity.com/paper/2602.18710/) (2026)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [CLEAR: Calibrated Learning for Epistemic and Aleatoric Risk](https://mlsanity.com/paper/2507.08150/) (2025)
- [Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithms](https://mlsanity.com/paper/2309.10301/) (2023)
- [Generative causal testing to bridge data-driven models and scientific theories in language neuroscience](https://mlsanity.com/paper/2410.00812/) (2024)
- [Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges](https://mlsanity.com/paper/2605.09702/) (2026)
- [Learning from learning machines: a new generation of AI technology to meet the needs of science](https://mlsanity.com/paper/2111.13786/) (2021)
- [Cluster Stability Selection](https://mlsanity.com/paper/2201.00494/) (2022)
- [KnowGraph: Knowledge-Enabled Anomaly Detection via Logical Reasoning on Graph Data](https://mlsanity.com/paper/2410.08390/) (2024)
- [PCS Workflow for Veridical Data Science in the Age of AI](https://mlsanity.com/paper/2508.00835/) (2025)
- [Sanity Checks for Agentic Data Science](https://mlsanity.com/paper/2604.11003/) (2026)
- [The Shapley Value of coalition of variables provides better explanations](https://mlsanity.com/paper/2103.13342/) (2021)

## Abstract (arXiv)

Building and expanding on principles of statistics, machine learning, and scientific inquiry, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework, comprised of both a workflow and documentation, aims to provide responsible, reliable, reproducible, and transparent results across the entire data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle for the data science life cycle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. Moreover, we develop inference procedures that build on PCS, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. We illustrate PCS inference through neuroscience and genomics projects of our own and others and compare it to existing methods in high dimensional, sparse linear model simulations. Over a wide range of misspecified simulation models, PCS inference demonstrates favorable performance in terms of ROC curves. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.