# Workflow Provenance in the Lifecycle of Scientific Machine Learning

arXiv: https://arxiv.org/abs/2010.00330 · HTML: https://mlsanity.com/paper/2010.00330/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2020.

## Built on by (most cited first, 9 of 9)

- [Management of Machine Learning Lifecycle Artifacts: A Survey](https://mlsanity.com/paper/2210.11831/) (2022)
- [LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology](https://mlsanity.com/paper/2509.13978/) (2025)
- [Towards Lightweight Data Integration using Multi-workflow Provenance and Data Observability](https://mlsanity.com/paper/2308.09004/) (2023)
- [Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment](https://mlsanity.com/paper/2510.14008/) (2025)
- [Provenance Tracking in Large-Scale Machine Learning Systems](https://mlsanity.com/paper/2507.01075/) (2025)
- [Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds](https://mlsanity.com/paper/2107.00187/) (2021)
- [Enhancing Transparency and Traceability in Healthcare AI: The AI Product Passport](https://mlsanity.com/paper/2512.13702/) (2025)
- [Gypscie: A Cross-Platform AI Artifact Management System](https://mlsanity.com/paper/2604.10311/) (2026)
- [Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data](https://mlsanity.com/paper/2608.11022/) (2026)

## Abstract (arXiv)

Machine Learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design principles to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the principles enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.