# ML-EXray: Visibility into ML Deployment on the Edge

arXiv: https://arxiv.org/abs/2111.04779 · HTML: https://mlsanity.com/paper/2111.04779/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 8 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems](https://mlsanity.com/paper/2010.08678/) (2020)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [MLPerf Training Benchmark](https://mlsanity.com/paper/1910.01500/) (2019)
- [On-Device Neural Net Inference with Mobile GPUs](https://mlsanity.com/paper/1907.01989/) (2019)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices](https://mlsanity.com/paper/2005.05085/) (2020)
- [Characterizing and Taming Model Instability Across Edge Devices](https://mlsanity.com/paper/2010.09028/) (2020)
- [On the quantization of recurrent neural networks](https://mlsanity.com/paper/2101.05453/) (2021)

## Built on by (most cited first, 6 of 6)

- [Monitoring and Adapting ML Models on Mobile Devices](https://mlsanity.com/paper/2305.07772/) (2023)
- [How to Manage Tiny Machine Learning at Scale: An Industrial Perspective](https://mlsanity.com/paper/2202.09113/) (2022)
- [U-TOE: Universal TinyML On-board Evaluation Toolkit for Low-Power IoT](https://mlsanity.com/paper/2306.14574/) (2023)
- [Datasheets for Machine Learning Sensors](https://mlsanity.com/paper/2306.08848/) (2023)
- [DEBUG-HD: Debugging TinyML models on-device using Hyper-Dimensional computing](https://mlsanity.com/paper/2411.10692/) (2024)
- [MGit: A Model Versioning and Management System](https://mlsanity.com/paper/2307.07507/) (2023)

## Abstract (arXiv)

Benefiting from expanding cloud infrastructure, deep neural networks (DNNs) today have increasingly high performance when trained in the cloud. Researchers spend months of effort competing for an extra few percentage points of model accuracy. However, when these models are actually deployed on edge devices in practice, very often, the performance can abruptly drop over 10% without obvious reasons. The key challenge is that there is not much visibility into ML inference execution on edge devices, and very little awareness of potential issues during the edge deployment process. We present ML-EXray, an end-to-end framework, which provides visibility into layer-level details of the ML execution, and helps developers analyze and debug cloud-to-edge deployment issues. More often than not, the reason for sub-optimal edge performance does not only lie in the model itself, but every operation throughout the data flow and the deployment process. Evaluations show that ML-EXray can effectively catch deployment issues, such as pre-processing bugs, quantization issues, suboptimal kernels, etc. Using ML-EXray, users need to write less than 15 lines of code to fully examine the edge deployment pipeline. Eradicating these issues, ML-EXray can correct model performance by up to 30%, pinpoint error-prone layers, and guide users to optimize kernel execution latency by two orders of magnitude. Code and APIs will be released as an open-source multi-lingual instrumentation library and a Python deployment validation library.