# Towards Lightweight Data Integration using Multi-workflow Provenance and Data Observability

arXiv: https://arxiv.org/abs/2308.09004 · HTML: https://mlsanity.com/paper/2308.09004/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Parsl: Pervasive Parallel Programming in Python](https://mlsanity.com/paper/1905.02158/) (2019)
- [AtomAI: A Deep Learning Framework for Analysis of Image and Spectroscopy Data in (Scanning) Transmission Electron Microscopy and Beyond](https://mlsanity.com/paper/2105.07485/) (2021)
- [Workflow Provenance in the Lifecycle of Scientific Machine Learning](https://mlsanity.com/paper/2010.00330/) (2020)
- [WfCommons: A Framework for Enabling Scientific Workflow Research and Development](https://mlsanity.com/paper/2105.14352/) (2021)
- [YesWorkflow: A User-Oriented, Language-Independent Tool for Recovering Workflow Information from Scripts](https://mlsanity.com/paper/1502.02403/) (2015)
- [Keeping Track of User Steering Actions in Dynamic Workflows](https://mlsanity.com/paper/1905.07167/) (2019)
- [Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds](https://mlsanity.com/paper/2107.00187/) (2021)
- [Workflows Community Summit 2022: A Roadmap Revolution](https://mlsanity.com/paper/2304.00019/) (2023)
- [ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum](https://mlsanity.com/paper/2307.10658/) (2023)
- [PROV-IO+: A Cross-Platform Provenance Framework for Scientific Data on HPC Systems](https://mlsanity.com/paper/2308.00891/) (2023)

## Built on by (most cited first, 8 of 8)

- [LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology](https://mlsanity.com/paper/2509.13978/) (2025)
- [Provenance Tracking in Large-Scale Machine Learning Systems](https://mlsanity.com/paper/2507.01075/) (2025)
- [yProv4ML: Effortless Provenance Tracking for Machine Learning Systems](https://mlsanity.com/paper/2507.01078/) (2025)
- [Data Readiness for Scientific AI at Scale](https://mlsanity.com/paper/2507.23018/) (2025)
- [Automated Data Readiness for Scientific AI](https://mlsanity.com/paper/2607.02771/) (2026)
- [An Agentic AI Framework to Accelerate Scientific Discovery in Plant Phenotyping](https://mlsanity.com/paper/2606.31831/) (2026)
- [SetGo: Metadata Readiness for Scientific AI Datasets](https://mlsanity.com/paper/2607.22677/) (2026)
- [Workflow Cards: Structured Summaries of Workflow Executions Using Provenance Data](https://mlsanity.com/paper/2608.11022/) (2026)

## Abstract (arXiv)

Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.