# COSTREAM: Learned Cost Models for Operator Placement in Edge-Cloud Environments

arXiv: https://arxiv.org/abs/2403.08444 · HTML: https://mlsanity.com/paper/2403.08444/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 47 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Learning Scheduling Algorithms for Data Processing Clusters](https://mlsanity.com/paper/1810.01963/) (2018)
- [Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings](https://mlsanity.com/paper/2002.12393/) (2020)
- [Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction](https://mlsanity.com/paper/2201.00561/) (2022)
- [Benchmarking Distributed Stream Data Processing Systems](https://mlsanity.com/paper/1802.08496/) (2018)
- [Model-Free Control for Distributed Stream Data Processing using Deep Reinforcement Learning](https://mlsanity.com/paper/1803.01016/) (2018)
- [Zero-Shot Cost Models for Distributed Stream Processing](https://mlsanity.com/paper/2207.03823/) (2022)
- [RIoTBench: A Real-time IoT Benchmark for Distributed Stream Processing Platforms](https://mlsanity.com/paper/1701.08530/) (2017)
- [R-Storm: Resource-Aware Scheduling in Storm](https://mlsanity.com/paper/1904.05456/) (2019)
- [Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning](https://mlsanity.com/paper/1911.08517/) (2019)
- [TCEP: Transitions in Operator Placement to Adapt to Dynamic Network Environments](https://mlsanity.com/paper/2106.12429/) (2021)

## Abstract (arXiv)

In this work, we present COSTREAM, a novel learned cost model for Distributed Stream Processing Systems that provides accurate predictions of the execution costs of a streaming query in an edge-cloud environment. The cost model can be used to find an initial placement of operators across heterogeneous hardware, which is particularly important in these environments. In our evaluation, we demonstrate that COSTREAM can produce highly accurate cost estimates for the initial operator placement and even generalize to unseen placements, queries, and hardware. When using COSTREAM to optimize the placements of streaming operators, a median speed-up of around 21x can be achieved compared to baselines.