# Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie

arXiv: https://arxiv.org/abs/2404.13682 · HTML: https://mlsanity.com/paper/2404.13682/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 6 of 6)

- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [Towards Scalable Dataframe Systems](https://mlsanity.com/paper/2001.00888/) (2020)
- [Building a serverless Data Lakehouse from spare parts](https://mlsanity.com/paper/2308.05368/) (2023)
- [Reasonable Scale Machine Learning with Open-Source Metaflow](https://mlsanity.com/paper/2303.11761/) (2023)
- [The Data Lakehouse: Data Warehousing and More](https://mlsanity.com/paper/2310.08697/) (2023)
- [A survey study of success factors in data science projects](https://mlsanity.com/paper/2201.06310/) (2022)

## Built on by (most cited first, 5 of 5)

- [Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehouse](https://mlsanity.com/paper/2510.09567/) (2025)
- [Building a Correct-by-Design Lakehouse. Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents](https://mlsanity.com/paper/2602.02335/) (2026)
- [Bauplan: zero-copy, scale-up FaaS for data pipelines](https://mlsanity.com/paper/2410.17465/) (2024)
- ["Skill Issues'': Data-Centric Optimization of Lakehouse Agents](https://mlsanity.com/paper/2606.01185/) (2026)
- [Querying Everything Everywhere All at Once: Supervaluationism for the Agentic Lakehouse](https://mlsanity.com/paper/2603.13380/) (2026)

## Abstract (arXiv)

As the Lakehouse architecture becomes more widespread, ensuring the reproducibility of data workloads over data lakes emerges as a crucial concern for data engineers. However, achieving reproducibility remains challenging. The size of data pipelines contributes to slow testing and iterations, while the intertwining of business logic and data management complicates debugging and increases error susceptibility. In this paper, we highlight recent advancements made at Bauplan in addressing this challenge. We introduce a system designed to decouple compute from data management, by leveraging a cloud runtime alongside Nessie, an open-source catalog with Git semantics. Demonstrating the system's capabilities, we showcase its ability to offer time-travel and branching semantics on top of object storage, and offer full pipeline reproducibility with a few CLI commands.