# Koji: Automating pipelines with mixed-semantics data sources

arXiv: https://arxiv.org/abs/1901.01908 · HTML: https://mlsanity.com/paper/1901.01908/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2018.

## Built on by (most cited first, 1 of 1)

- [SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Science](https://mlsanity.com/paper/2503.23314/) (2025)

## Abstract (arXiv)

We propose a new result-oriented semantic for defining data processing workflows that manipulate data in different semantic forms (files or services) in a unified manner. This approach enables users to define workflows for a vast variety of reproducible data-processing tasks in a simple declarative manner which focuses on application-level results, while automating all control-plane considerations (like failure recovery without loss of progress and computation reuse) behind the scenes. The uniform treatment of files and services as data enables easy integration with existing data sources (e.g. data acquisition APIs) and sinks of data (e.g. database services). Whereas the focus on containers as transformations enables reuse of existing data-processing systems. We describe a declarative configuration mechanism, which can be viewed as an intermediate representation (IR) of reproducible data processing pipelines in the same spirit as, for instance, TensorFlowcitetensorflow and ONNXciteonnx utilize IRs for defining tensor-processing pipelines.