# A Data Source Dependency Analysis Framework for Large Scale Data Science Projects

arXiv: https://arxiv.org/abs/2212.07951 · HTML: https://mlsanity.com/paper/2212.07951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 15 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 3 of 3)

- [MLOps -- Definitions, Tools and Challenges](https://mlsanity.com/paper/2201.00162/) (2022)
- [A Static Analysis Framework for Data Science Notebooks](https://mlsanity.com/paper/2110.08339/) (2021)
- [Abstract Interpretation-Based Data Leakage Static Analysis](https://mlsanity.com/paper/2211.16073/) (2022)

## Abstract (arXiv)

Dependency hell is a well-known pain point in the development of large software projects and machine learning (ML) code bases are not immune from it. In fact, ML applications suffer from an additional form, namely, "data source dependency hell". This term refers to the central role played by data and its unique quirks that often lead to unexpected failures of ML models which cannot be explained by code changes. In this paper, we present an automated dependency mapping framework that allows MLOps engineers to monitor the whole dependency map of their models in a fast paced engineering environment and thus mitigate ahead of time the consequences of any data source changes (e.g., re-train model, ignore data, set default data etc.). Our system is based on a unified and generic approach, employing techniques from static analysis, from which data sources can be identified reliably for any type of dependency on a wide range of source languages and artefacts. The dependency mapping framework is exposed as a REST web API where the only input is the path to the Git repository hosting the code base. Currently used by MLOps engineers at Microsoft, we expect such dependency map APIs to be adopted more widely by MLOps engineers in the future.