# Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps

arXiv: https://arxiv.org/abs/2002.07687 · HTML: https://mlsanity.com/paper/2002.07687/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 30 of 36)

- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Smart at what cost? Characterising Mobile Deep Neural Networks in the wild](https://mlsanity.com/paper/2109.13963/) (2021)
- [No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML](https://mlsanity.com/paper/2310.07152/) (2023)
- [Robustness of on-device Models: Adversarial Attack to Deep Learning Models on Android Apps](https://mlsanity.com/paper/2101.04401/) (2021)
- [NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation](https://mlsanity.com/paper/2305.00097/) (2023)
- [SLIP: Securing LLMs IP Using Weights Decomposition](https://mlsanity.com/paper/2407.10886/) (2024)
- [Smart App Attack: Hacking Deep Learning Models in Android Apps](https://mlsanity.com/paper/2204.11075/) (2022)
- [Octopus v4: Graph of language models](https://mlsanity.com/paper/2404.19296/) (2024)
- [Efficient Query-Based Attack against ML-Based Android Malware Detection under Zero Knowledge Setting](https://mlsanity.com/paper/2309.01866/) (2023)
- [Confidential Machine Learning Computation in Untrusted Environments: A Systems Security Perspective](https://mlsanity.com/paper/2111.03308/) (2021)
- [A Light-weight Deep Learning Model for Remote Sensing Image Classification](https://mlsanity.com/paper/2302.13028/) (2023)
- [Edge Security: Challenges and Issues](https://mlsanity.com/paper/2206.07164/) (2022)
- [A Robust and Low Complexity Deep Learning Model for Remote Sensing Image Classification](https://mlsanity.com/paper/2211.02820/) (2022)
- [InferNet: Exploiting Aggregate GPU Profiles as Side-Channel for DNN Architecture Inference](https://mlsanity.com/paper/2304.03388/) (2023)
- [TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models](https://mlsanity.com/paper/2411.09945/) (2024)
- [Towards Data Governance of Frontier AI Models](https://mlsanity.com/paper/2412.03824/) (2024)
- [DeepTaster: Adversarial Perturbation-Based Fingerprinting to Identify Proprietary Dataset Use in Deep Neural Networks](https://mlsanity.com/paper/2211.13535/) (2022)
- [Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2305.03963/) (2023)
- [Neural Dehydration: Effective Erasure of Black-box Watermarks from DNNs with Limited Data](https://mlsanity.com/paper/2309.03466/) (2023)
- [A Picture is Worth 500 Labels: A Case Study of Demographic Disparities in Local Machine Learning Models for Instagram and TikTok](https://mlsanity.com/paper/2403.19717/) (2024)
- [TBNet: A Neural Architectural Defense Framework Facilitating DNN Model Protection in Trusted Execution Environments](https://mlsanity.com/paper/2405.03974/) (2024)
- [THEMIS: Towards Practical Intellectual Property Protection for Post-Deployment On-Device Deep Learning Models](https://mlsanity.com/paper/2503.23748/) (2025)
- [Robust, General, and Low Complexity Acoustic Scene Classification Systems and An Effective Visualization for Presenting a Sound Scene Context](https://mlsanity.com/paper/2210.08610/) (2022)
- [An Empirical Study of AI Techniques in Mobile Applications](https://mlsanity.com/paper/2212.01635/) (2022)
- [Stealthy Backdoor Attack to Real-world Models in Android Apps](https://mlsanity.com/paper/2501.01263/) (2025)
- ["Impressively Scary:" Exploring User Perceptions and Reactions to Unraveling Machine Learning Models in Social Media Applications](https://mlsanity.com/paper/2503.03927/) (2025)
- [WGLE:Backdoor-free and Multi-bit Black-box Watermarking for Graph Neural Networks](https://mlsanity.com/paper/2506.08602/) (2025)
- [Evil from Within: Machine Learning Backdoors through Hardware Trojans](https://mlsanity.com/paper/2304.08411/) (2023)
- [Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps](https://mlsanity.com/paper/2307.00756/) (2023)
- [A Novel Access Control and Privacy-Enhancing Approach for Models in Edge Computing](https://mlsanity.com/paper/2411.03847/) (2024)

## Abstract (arXiv)

On-device machine learning (ML) is quickly gaining popularity among mobile apps. It allows offline model inference while preserving user privacy. However, ML models, considered as core intellectual properties of model owners, are now stored on billions of untrusted devices and subject to potential thefts. Leaked models can cause both severe financial loss and security consequences. This paper presents the first empirical study of ML model protection on mobile devices. Our study aims to answer three open questions with quantitative evidence: How widely is model protection used in apps? How robust are existing model protection techniques? What impacts can (stolen) models incur? To that end, we built a simple app analysis pipeline and analyzed 46,753 popular apps collected from the US and Chinese app markets. We identified 1,468 ML apps spanning all popular app categories. We found that, alarmingly, 41% of ML apps do not protect their models at all, which can be trivially stolen from app packages. Even for those apps that use model protection or encryption, we were able to extract the models from 66% of them via unsophisticated dynamic analysis techniques. The extracted models are mostly commercial products and used for face recognition, liveness detection, ID/bank card recognition, and malware detection. We quantitatively estimated the potential financial and security impact of a leaked model, which can amount to millions of dollars for different stakeholders. Our study reveals that on-device models are currently at high risk of being leaked; attackers are highly motivated to steal such models. Drawn from our large-scale study, we report our insights into this emerging security problem and discuss the technical challenges, hoping to inspire future research on robust and practical model protection for mobile devices.