# FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

arXiv: https://arxiv.org/abs/2603.09046 · HTML: https://mlsanity.com/paper/2603.09046/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 23 of its 80 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://mlsanity.com/paper/2309.11436/) (2023)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [The rising costs of training frontier AI models](https://mlsanity.com/paper/2405.21015/) (2024)
- [DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments](https://mlsanity.com/paper/2004.05703/) (2020)
- [Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms](https://mlsanity.com/paper/2410.18967/) (2024)
- [Apple Intelligence Foundation Language Models: Tech Report 2025](https://mlsanity.com/paper/2507.13575/) (2025)
- [Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference](https://mlsanity.com/paper/2501.14794/) (2025)
- [Position: The Most Expensive Part of an LLM should be its Training Data](https://mlsanity.com/paper/2504.12427/) (2025)
- [Serdab: An IoT Framework for Partitioning Neural Networks Computation across Multiple Enclaves](https://mlsanity.com/paper/2005.06043/) (2020)
- [TransLinkGuard: Safeguarding Transformer Models Against Model Stealing in Edge Deployment](https://mlsanity.com/paper/2404.11121/) (2024)
- [MobiAgent: A Systematic Framework for Customizable Mobile Agents](https://mlsanity.com/paper/2509.00531/) (2025)
- [UI-UG: A Unified MLLM for UI Understanding and Generation](https://mlsanity.com/paper/2509.24361/) (2025)
- [TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone](https://mlsanity.com/paper/2511.13717/) (2025)
- [Robust Safe Reinforcement Learning under Adversarial Disturbances](https://mlsanity.com/paper/2310.07207/) (2023)
- [TrustShadow: Secure Execution of Unmodified Applications with ARM TrustZone](https://mlsanity.com/paper/1704.05600/) (2017)

## Built on by (most cited first, 1 of 1)

- [SoK: Attack and Defense Landscape of Mobile On-device AI Systems](https://mlsanity.com/paper/2607.00362/) (2026)

## Abstract (arXiv)

Device-side Large Language Models (LLMs) have witnessed explosive growth, offering higher privacy and availability compared to cloud-side LLMs. During LLM inference, both model weights and user data are valuable, and attackers may even compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead due to its inflexible isolation of memory and the NPU. To address these challenges, this paper introduces FlexServe, a fast and secure LLM serving system for mobile devices. It first introduces a Flexible Resource Isolation mechanism to construct Flexible Secure Memory (Flex-Mem) and Flexible Secure NPU (Flex-NPU). Both memory pages and the NPU can be efficiently switched between unprotected and protected modes. Based on these mechanisms, FlexServe designs a fast and secure LLM inference framework within TrustZone's secure world. The LLM-Aware Memory Management and Secure Inference Pipeline are introduced to accelerate inference. A Multi-Model Scheduler is proposed to optimize multi-model workflows. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves an average 10.05× speedup in Time to First Token (TTFT) compared to the strawman, and an average 2.44× TTFT speedup compared to an optimized strawman with pipeline and secure NPU enabled. For multi-model agent workflows, the end-to-end speedup is up to 24.30× and 4.05× compared to the strawman and optimized strawman, respectively.