# Understanding Real-world Threats to Deep Learning Models in Android Apps

arXiv: https://arxiv.org/abs/2209.09577 · HTML: https://mlsanity.com/paper/2209.09577/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2022.

## Built on by (most cited first, 15 of 15)

- [No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML](https://mlsanity.com/paper/2310.07152/) (2023)
- [Synthetic is all you need: removing the auxiliary data assumption for membership inference attacks against synthetic data](https://mlsanity.com/paper/2307.01701/) (2023)
- [Investigating the Effect of Misalignment on Membership Privacy in the White-box Setting](https://mlsanity.com/paper/2306.05093/) (2023)
- [On the Robustness of Dataset Inference](https://mlsanity.com/paper/2210.13631/) (2022)
- [Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2305.03963/) (2023)
- [A Picture is Worth 500 Labels: A Case Study of Demographic Disparities in Local Machine Learning Models for Instagram and TikTok](https://mlsanity.com/paper/2403.19717/) (2024)
- [Principles of Designing Robust Remote Face Anti-Spoofing Systems](https://mlsanity.com/paper/2406.03684/) (2024)
- [THEMIS: Towards Practical Intellectual Property Protection for Post-Deployment On-Device Deep Learning Models](https://mlsanity.com/paper/2503.23748/) (2025)
- [An Empirical Study of AI Techniques in Mobile Applications](https://mlsanity.com/paper/2212.01635/) (2022)
- [Stealthy Backdoor Attack to Real-world Models in Android Apps](https://mlsanity.com/paper/2501.01263/) (2025)
- ["Impressively Scary:" Exploring User Perceptions and Reactions to Unraveling Machine Learning Models in Social Media Applications](https://mlsanity.com/paper/2503.03927/) (2025)
- [Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps](https://mlsanity.com/paper/2307.00756/) (2023)
- [Lorica: A Synergistic Fine-Tuning Framework for Advancing Personalized Adversarial Robustness](https://mlsanity.com/paper/2506.05402/) (2025)
- [NeuroDeX: Unlocking Diverse Support in Decompiling Deep Neural Network Executables](https://mlsanity.com/paper/2509.06402/) (2025)
- [SoK: Attack and Defense Landscape of Mobile On-device AI Systems](https://mlsanity.com/paper/2607.00362/) (2026)

## Abstract (arXiv)

Famous for its superior performance, deep learning (DL) has been popularly used within many applications, which also at the same time attracts various threats to the models. One primary threat is from adversarial attacks. Researchers have intensively studied this threat for several years and proposed dozens of approaches to create adversarial examples (AEs). But most of the approaches are only evaluated on limited models and datasets (e.g., MNIST, CIFAR-10). Thus, the effectiveness of attacking real-world DL models is not quite clear. In this paper, we perform the first systematic study of adversarial attacks on real-world DNN models and provide a real-world model dataset named RWM. Particularly, we design a suite of approaches to adapt current AE generation algorithms to the diverse real-world DL models, including automatically extracting DL models from Android apps, capturing the inputs and outputs of the DL models in apps, generating AEs and validating them by observing the apps' execution. For black-box DL models, we design a semantic-based approach to build suitable datasets and use them for training substitute models when performing transfer-based attacks. After analyzing 245 DL models collected from 62,583 real-world apps, we have a unique opportunity to understand the gap between real-world DL models and contemporary AE generation algorithms. To our surprise, the current AE generation algorithms can only directly attack 6.53% of the models. Benefiting from our approach, the success rate upgrades to 47.35%.