# RIBBON: Cost-Effective and QoS-Aware Deep Learning Model Inference using a Diverse Pool of Cloud Computing Instances

arXiv: https://arxiv.org/abs/2207.11434 · HTML: https://mlsanity.com/paper/2207.11434/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 97 references that are in the map.
- First posted on arXiv 23 Jul 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Wide & Deep Learning for Recommender Systems](https://mlsanity.com/paper/1606.07792/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep learning for universal linear embeddings of nonlinear dynamics](https://mlsanity.com/paper/1712.09707/) (2017)
- [Deep Interest Evolution Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1809.03672/) (2018)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Deep Learning vs. Traditional Computer Vision](https://mlsanity.com/paper/1910.13796/) (2019)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [Deep Learning COVID-19 Features on CXR using Limited Training Data Sets](https://mlsanity.com/paper/2004.05758/) (2020)
- [Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes](https://mlsanity.com/paper/1706.03673/) (2017)
- [DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference](https://mlsanity.com/paper/2001.02772/) (2020)
- [Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems](https://mlsanity.com/paper/2003.09518/) (2020)
- [Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications](https://mlsanity.com/paper/1811.09886/) (2018)
- [DLHub: Model and Data Serving for Science](https://mlsanity.com/paper/1811.11213/) (2018)
- [Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures](https://mlsanity.com/paper/2005.04680/) (2020)
- [Multi-objective Bayesian optimisation with preferences over objectives](https://mlsanity.com/paper/1902.04228/) (2019)
- [Cross-Stack Workload Characterization of Deep Recommendation Systems](https://mlsanity.com/paper/2010.05037/) (2020)
- [Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM](https://mlsanity.com/paper/1712.10081/) (2017)
- [Bayesian Optimization for Multi-objective Optimization and Multi-point Search](https://mlsanity.com/paper/1905.02370/) (2019)
- [Micky: A Cheaper Alternative for Selecting Cloud Instances](https://mlsanity.com/paper/1803.05587/) (2018)
- [Performance Analysis and Modeling of Video Transcoding Using Heterogeneous Cloud Services](https://mlsanity.com/paper/1809.06529/) (2018)

## Built on by (most cited first, 1 of 1)

- [KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources](https://mlsanity.com/paper/2210.05889/) (2022)

## Abstract (arXiv)

Deep learning model inference is a key service in many businesses and scientific discovery processes. This paper introduces RIBBON, a novel deep learning inference serving system that meets two competing objectives: quality-of-service (QoS) target and cost-effectiveness. The key idea behind RIBBON is to intelligently employ a diverse set of cloud computing instances (heterogeneous instances) to meet the QoS target and maximize cost savings. RIBBON devises a Bayesian Optimization-driven strategy that helps users build the optimal set of heterogeneous instances for their model inference service needs on cloud computing platforms -- and, RIBBON demonstrates its superiority over existing approaches of inference serving systems using homogeneous instance pools. RIBBON saves up to 16% of the inference service cost for different learning models including emerging deep learning recommender system models and drug-discovery enabling models.