# SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle

arXiv: https://arxiv.org/abs/1909.02976 · HTML: https://mlsanity.com/paper/1909.02976/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 20 of 20)

- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale](https://mlsanity.com/paper/2201.06834/) (2022)
- [Management of Machine Learning Lifecycle Artifacts: A Survey](https://mlsanity.com/paper/2210.11831/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)
- [SubStrat: A Subset-Based Strategy for Faster AutoML](https://mlsanity.com/paper/2206.03070/) (2022)
- [AutoML in Heavily Constrained Applications](https://mlsanity.com/paper/2306.16913/) (2023)
- [Saturn: An Optimized Data System for Large Model Deep Learning Workloads](https://mlsanity.com/paper/2309.01226/) (2023)
- [The Duck's Brain: Training and Inference of Neural Networks in Modern Database Engines](https://mlsanity.com/paper/2312.17355/) (2023)
- [stratum: A System Infrastructure for Massive Agent-Centric ML Workloads](https://mlsanity.com/paper/2603.03589/) (2026)
- [Efficient End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2206.09423/) (2022)
- [MaskSearch: Querying Image Masks at Scale](https://mlsanity.com/paper/2305.02375/) (2023)
- [A Declarative Query Language for Scientific Machine Learning](https://mlsanity.com/paper/2405.16159/) (2024)
- [Morphing-based Compression for Data-centric ML Pipelines](https://mlsanity.com/paper/2504.11067/) (2025)
- [InferF: Declarative Factorization of AI/ML Inferences over Joins](https://mlsanity.com/paper/2511.20489/) (2025)
- [Automated Tensor-Relational Decomposition for Large-Scale Sparse Tensor Computation](https://mlsanity.com/paper/2603.08957/) (2026)
- [Gypscie: A Cross-Platform AI Artifact Management System](https://mlsanity.com/paper/2604.10311/) (2026)
- [A Typed Tensor Language for Shared-State Federated Computation](https://mlsanity.com/paper/2605.21103/) (2026)
- [FlowPipe: LLM-Enhanced Conditional Generative Flow Networks for Data Preparation Pipeline Construction](https://mlsanity.com/paper/2606.24679/) (2026)
- [ML Engineering Agents - A Data Management Perspective](https://mlsanity.com/paper/2610.16919/) (2026)

## Abstract (arXiv)

Machine learning (ML) applications become increasingly common in many domains. ML systems to execute these workloads include numerical computing frameworks and libraries, ML algorithm libraries, and specialized systems for deep neural networks and distributed ML. These systems focus primarily on efficient model training and scoring. However, the data science process is exploratory, and deals with underspecified objectives and a wide variety of heterogeneous data sources. Therefore, additional tools are employed for data engineering and debugging, which requires boundary crossing, unnecessary manual effort, and lacks optimization across the lifecycle. In this paper, we introduce SystemDS, an open source ML system for the end-to-end data science lifecycle from data integration, cleaning, and preparation, over local, distributed, and federated ML model training, to debugging and serving. To this end, we aim to provide a stack of declarative language abstractions for the different lifecycle tasks, and users with different expertise. We describe the overall system architecture, explain major design decisions (motivated by lessons learned from Apache SystemML), and discuss key features and research directions. Finally, we provide preliminary results that show the potential of end-to-end lifecycle optimization.