# ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2011.05905 · HTML: https://mlsanity.com/paper/2011.05905/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2020.

## Built on by (most cited first, 21 of 21)

- [NNSplitter: An Active Defense Solution for DNN Model via Automated Weight Obfuscation](https://mlsanity.com/paper/2305.00097/) (2023)
- [DVFO: Learning-Based DVFS for Energy-Efficient Edge-Cloud Collaborative Inference](https://mlsanity.com/paper/2306.01811/) (2023)
- [Re-Key-Free, Risky-Free: Adaptable Model Usage Control](https://mlsanity.com/paper/2511.18772/) (2025)
- [CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment](https://mlsanity.com/paper/2410.13903/) (2024)
- [Hyperdimensional Computing as a Rescue for Efficient Privacy-Preserving Machine Learning-as-a-Service](https://mlsanity.com/paper/2310.06840/) (2023)
- [TBNet: A Neural Architectural Defense Framework Facilitating DNN Model Protection in Trusted Execution Environments](https://mlsanity.com/paper/2405.03974/) (2024)
- [SecureInfer: Heterogeneous TEE-GPU Architecture for Privacy-Critical Tensors for Large Language Model Deployment](https://mlsanity.com/paper/2510.19979/) (2025)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Staining and locking computer vision models without retraining](https://mlsanity.com/paper/2507.22000/) (2025)
- [Towards Confidential and Efficient LLM Inference with Dual Privacy Protection](https://mlsanity.com/paper/2509.09091/) (2025)
- [ProDiF: Protecting Domain-Invariant Features to Secure Pre-Trained Models Against Extraction](https://mlsanity.com/paper/2503.13224/) (2025)
- [Slice or the Whole Pie? Utility Control for AI Models](https://mlsanity.com/paper/2508.06551/) (2025)
- [DistilLock: Safeguarding LLMs from Unauthorized Knowledge Distillation on the Edge](https://mlsanity.com/paper/2510.16716/) (2025)
- [Good-Enough LLM Obfuscation (GELO)](https://mlsanity.com/paper/2603.05035/) (2026)
- [When Agents Handle Secrets: A Survey of Confidential Computing for Agentic AI](https://mlsanity.com/paper/2605.03213/) (2026)
- [LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters](https://mlsanity.com/paper/2605.13163/) (2026)
- [Communication-Efficient Verifiable Attention for LLM Inference](https://mlsanity.com/paper/2606.16352/) (2026)
- [SoK: Attack and Defense Landscape of Mobile On-device AI Systems](https://mlsanity.com/paper/2607.00362/) (2026)
- [SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels](https://mlsanity.com/paper/2608.02995/) (2026)
- [TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge](https://mlsanity.com/paper/2608.22716/) (2026)
- [From Acoustic Sensing to Physical Actions: A Comprehensive Survey of Security Risks in Edge-Deployed Audio LLM-Based Agents](https://mlsanity.com/paper/2610.15390/) (2026)

## Abstract (arXiv)

With the increased usage of AI accelerators on mobile and edge devices, on-device machine learning (ML) is gaining popularity. Thousands of proprietary ML models are being deployed today on billions of untrusted devices. This raises serious security concerns about model privacy. However, protecting model privacy without losing access to the untrusted AI accelerators is a challenging problem. In this paper, we present a novel on-device model inference system, ShadowNet. ShadowNet protects the model privacy with Trusted Execution Environment (TEE) while securely outsourcing the heavy linear layers of the model to the untrusted hardware accelerators. ShadowNet achieves this by transforming the weights of the linear layers before outsourcing them and restoring the results inside the TEE. The non-linear layers are also kept secure inside the TEE. ShadowNet's design ensures efficient transformation of the weights and the subsequent restoration of the results. We build a ShadowNet prototype based on TensorFlow Lite and evaluate it on five popular CNNs, namely, MobileNet, ResNet-44, MiniVGG, ResNet-404, and YOLOv4-tiny. Our evaluation shows that ShadowNet achieves strong security guarantees with reasonable performance, offering a practical solution for secure on-device model inference.