# Blink: Lightweight Sample Runs for Cost Optimization of Big Data Applications

arXiv: https://arxiv.org/abs/2207.02290 · HTML: https://mlsanity.com/paper/2207.02290/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2022.

## Built on by (most cited first, 1 of 1)

- [Karasu: A Collaborative Approach to Efficient Cluster Configuration for Big Data Analytics](https://mlsanity.com/paper/2308.11792/) (2023)

## Abstract (arXiv)

Distributed in-memory data processing engines accelerate iterative applications by caching substantial datasets in memory rather than recomputing them in each iteration. Selecting a suitable cluster size for caching these datasets plays an essential role in achieving optimal performance. In practice, this is a tedious and hard task for end users, who are typically not aware of cluster specifications, workload semantics and sizes of intermediate data. We present Blink, an autonomous sampling-based framework, which predicts sizes of cached datasets and selects optimal cluster size without relying on historical runs. We evaluate Blink on a variety of iterative, real-world, machine learning applications. With an average sample runs cost of 4.6% compared to the cost of optimal runs, Blink selects the optimal cluster size in 15 out of 16 cases, saving up to 47.4% of execution cost compared to average costs.