# Reproducible and Portable Big Data Analytics in the Cloud

arXiv: https://arxiv.org/abs/2112.09762 · HTML: https://mlsanity.com/paper/2112.09762/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 61 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 5 of 5)

- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Parsl: Pervasive Parallel Programming in Python](https://mlsanity.com/paper/1905.02158/) (2019)
- [An introduction to Docker for reproducible research, with examples from the R environment](https://mlsanity.com/paper/1410.0846/) (2014)
- [Terminologies for Reproducible Research](https://mlsanity.com/paper/1802.03311/) (2018)
- [Reproducible Experiments for Comparing Apache Flink and Apache Spark on Public Clouds](https://mlsanity.com/paper/1610.04493/) (2016)

## Built on by (most cited first, 1 of 1)

- [YOLO based Ocean Eddy Localization with AWS SageMaker](https://mlsanity.com/paper/2404.06744/) (2024)

## Abstract (arXiv)

Cloud computing has become a major approach to help reproduce computational experiments. Yet there are still two main difficulties in reproducing batch based big data analytics (including descriptive and predictive analytics) in the cloud. The first is how to automate end-to-end scalable execution of analytics including distributed environment provisioning, analytics pipeline description, parallel execution, and resource termination. The second is that an application developed for one cloud is difficult to be reproduced in another cloud, a.k.a. vendor lock-in problem. To tackle these problems, we leverage serverless computing and containerization techniques for automated scalable execution and reproducibility, and utilize the adapter design pattern to enable application portability and reproducibility across different clouds. We propose and develop an open-source toolkit that supports 1) fully automated end-to-end execution and reproduction via a single command, 2) automated data and configuration storage for each execution, 3) flexible client modes based on user preferences, 4) execution history query, and 5) simple reproduction of existing executions in the same environment or a different environment. We did extensive experiments on both AWS and Azure using four big data analytics applications that run on virtual CPU/GPU clusters. The experiments show our toolkit can achieve good execution performance, scalability, and efficient reproducibility for cloud-based big data analytics.