# Characterizing and Taming Model Instability Across Edge Devices

arXiv: https://arxiv.org/abs/2010.09028 · HTML: https://mlsanity.com/paper/2010.09028/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [Neural Network Verification is a Programming Language Challenge](https://mlsanity.com/paper/2501.05867/) (2025)
- [ML-EXray: Visibility into ML Deployment on the Edge](https://mlsanity.com/paper/2111.04779/) (2021)
- [A Tale of Two Models: Constructing Evasive Attacks on Edge Models](https://mlsanity.com/paper/2204.10933/) (2022)
- [Monitoring and Adapting ML Models on Mobile Devices](https://mlsanity.com/paper/2305.07772/) (2023)
- [HeteroSwitch: Characterizing and Taming System-Induced Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2403.04207/) (2024)
- [DEBUG-HD: Debugging TinyML models on-device using Hyper-Dimensional computing](https://mlsanity.com/paper/2411.10692/) (2024)
- [Edge AI Deployment Beyond Models: A BSP-Aware Systems Framework for Industrial Embedded Platforms](https://mlsanity.com/paper/2605.26119/) (2026)

## Abstract (arXiv)

The same machine learning model running on different edge devices may produce highly-divergent outputs on a nearly-identical input. Possible reasons for the divergence include differences in the device sensors, the device's signal processing hardware and software, and its operating system and processors. This paper presents the first methodical characterization of the variations in model prediction across real-world mobile devices. We demonstrate that accuracy is not a useful metric to characterize prediction divergence, and introduce a new metric, instability, which captures this variation. We characterize different sources for instability, and show that differences in compression formats and image signal processing account for significant instability in object classification models. Notably, in our experiments, 14-17% of images produced divergent classifications across one or more phone models. We evaluate three different techniques for reducing instability. In particular, we adapt prior work on making models robust to noise in order to fine-tune models to be robust to variations across edge devices. We demonstrate our fine-tuning techniques reduce instability by 75%.