# Automatic and Efficient Customization of Neural Networks for ML Applications

arXiv: https://arxiv.org/abs/2310.04685 · HTML: https://mlsanity.com/paper/2310.04685/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 75 references that are in the map.
- First posted on arXiv 7 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [Serving DNNs like Clockwork: Performance Predictability from the Bottom Up](https://mlsanity.com/paper/2006.02464/) (2020)
- [Online Model Distillation for Efficient Video Inference](https://mlsanity.com/paper/1812.02699/) (2018)
- [BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics](https://mlsanity.com/paper/1805.01046/) (2018)
- [Google's Cloud Vision API Is Not Robust To Noise](https://mlsanity.com/paper/1704.05051/) (2017)
- [FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply](https://mlsanity.com/paper/2006.07512/) (2020)
- [GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge](https://mlsanity.com/paper/2201.07705/) (2022)
- [PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems](https://mlsanity.com/paper/1810.06115/) (2018)
- [Physical Representation-based Predicate Optimization for a Visual Analytics Database](https://mlsanity.com/paper/1806.04226/) (2018)
- [Fast Video Classification via Adaptive Cascading of Deep Models](https://mlsanity.com/paper/1611.06453/) (2016)
- [Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications](https://mlsanity.com/paper/1902.04610/) (2019)
- [Did the Model Change? Efficiently Assessing Machine Learning API Shifts](https://mlsanity.com/paper/2107.14203/) (2021)
- [THIA: Accelerating Video Analytics using Early Inference and Fine-Grained Query Planning](https://mlsanity.com/paper/2102.08481/) (2021)
- [Video Monitoring Queries](https://mlsanity.com/paper/2002.10537/) (2020)
- [Optimization Beyond the Convolution: Generalizing Spatial Relations with End-to-End Metric Learning](https://mlsanity.com/paper/1707.00893/) (2017)
- [Cost Effective MLaaS Federation: A Combinatorial Reinforcement Learning Approach](https://mlsanity.com/paper/2204.13971/) (2022)
- [A Study of Data Pre-processing Techniques for Imbalanced Biomedical Data Classification](https://mlsanity.com/paper/1911.00996/) (2019)
- [Pool of Experts: Realtime Querying Specialized Knowledge in Massive Neural Networks](https://mlsanity.com/paper/2107.01354/) (2021)

## Abstract (arXiv)

ML APIs have greatly relieved application developers of the burden to design and train their own neural network models -- classifying objects in an image can now be as simple as one line of Python code to call an API. However, these APIs offer the same pre-trained models regardless of how their output is used by different applications. This can be suboptimal as not all ML inference errors can cause application failures, and the distinction between inference errors that can or cannot cause failures varies greatly across applications. To tackle this problem, we first study 77 real-world applications, which collectively use six ML APIs from two providers, to reveal common patterns of how ML API output affects applications' decision processes. Inspired by the findings, we propose ChameleonAPI, an optimization framework for ML APIs, which takes effect without changing the application source code. ChameleonAPI provides application developers with a parser that automatically analyzes the application to produce an abstract of its decision process, which is then used to devise an application-specific loss function that only penalizes API output errors critical to the application. ChameleonAPI uses the loss function to efficiently train a neural network model customized for each application and deploys it to serve API invocations from the respective application via existing interface. Compared to a baseline that selects the best-of-all commercial ML API, we show that ChameleonAPI reduces incorrect application decisions by 43%.