# Towards better data discovery and collection with flow-based programming

arXiv: https://arxiv.org/abs/2108.04105 · HTML: https://mlsanity.com/paper/2108.04105/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 9 Aug 2021.

## Builds on (most cited first, 7 of 7)

- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [MLlib: Machine Learning in Apache Spark](https://mlsanity.com/paper/1505.06807/) (2015)
- [Continual Learning in Practice](https://mlsanity.com/paper/1903.05202/) (2019)
- [Data Readiness Levels](https://mlsanity.com/paper/1705.02245/) (2017)
- [Data Engineering for Data Analytics: A Classification of the Issues, and Case Studies](https://mlsanity.com/paper/2004.12929/) (2020)
- [Data-driven Workflows for Microservices](https://mlsanity.com/paper/1511.02597/) (2015)
- [Long-Term Evaluation of Technical Debt in Open-Source Software](https://mlsanity.com/paper/2007.13422/) (2020)

## Built on by (most cited first, 3 of 3)

- [An Empirical Evaluation of Flow Based Programming in the Machine Learning Deployment Context](https://mlsanity.com/paper/2204.12781/) (2022)
- [Causal fault localisation in dataflow systems](https://mlsanity.com/paper/2304.11987/) (2023)
- [Automated Planning for Optimal Data Pipeline Instantiation](https://mlsanity.com/paper/2503.12626/) (2025)

## Abstract (arXiv)

Despite huge successes reported by the field of machine learning, such as voice assistants or self-driving cars, businesses still observe very high failure rate when it comes to deployment of ML in production. We argue that part of the reason is infrastructure that was not designed for data-oriented activities. This paper explores the potential of flow-based programming (FBP) for simplifying data discovery and collection in software systems. We compare FBP with the currently prevalent service-oriented paradigm to assess characteristics of each paradigm in the context of ML deployment. We develop a data processing application, formulate a subsequent ML deployment task, and measure the impact of the task implementation within both programming paradigms. Our main conclusion is that FBP shows great potential for providing data-centric infrastructural benefits for deployment of ML. Additionally, we provide an insight into the current trend that prioritizes model development over data quality management.