# SecureInfer: Heterogeneous TEE-GPU Architecture for Privacy-Critical Tensors for Large Language Model Deployment

arXiv: https://arxiv.org/abs/2510.19979 · HTML: https://mlsanity.com/paper/2510.19979/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Knockoff Nets: Stealing Functionality of Black-Box Models](https://mlsanity.com/paper/1812.02766/) (2018)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models](https://mlsanity.com/paper/2102.02551/) (2021)
- [DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments](https://mlsanity.com/paper/2004.05703/) (2020)
- [ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks](https://mlsanity.com/paper/2011.05905/) (2020)
- [No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML](https://mlsanity.com/paper/2310.07152/) (2023)
- [Depth Gives a False Sense of Privacy: LLM Internal States Inversion](https://mlsanity.com/paper/2507.16372/) (2025)
- [InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks](https://mlsanity.com/paper/2411.18191/) (2024)
- [TMI! Finetuned Models Leak Private Information from their Pretraining Data](https://mlsanity.com/paper/2306.01181/) (2023)
- [Serdab: An IoT Framework for Partitioning Neural Networks Computation across Multiple Enclaves](https://mlsanity.com/paper/2005.06043/) (2020)
- ["Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillation](https://mlsanity.com/paper/2409.02718/) (2024)
- [Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices](https://mlsanity.com/paper/2503.14932/) (2025)

## Built on by (most cited first, 2 of 2)

- [Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks](https://mlsanity.com/paper/2602.16760/) (2026)
- [TEE-X: TEE-aware Acceleration Framework for Large Vision Models at the Edge](https://mlsanity.com/paper/2608.22716/) (2026)

## Abstract (arXiv)

With the increasing deployment of Large Language Models (LLMs) on mobile and edge platforms, securing them against model extraction attacks has become a pressing concern. However, protecting model privacy without sacrificing the performance benefits of untrusted AI accelerators, such as GPUs, presents a challenging trade-off. In this paper, we initiate the study of high-performance execution on LLMs and present SecureInfer, a hybrid framework that leverages a heterogeneous Trusted Execution Environments (TEEs)-GPU architecture to isolate privacy-critical components while offloading compute-intensive operations to untrusted accelerators. Building upon an outsourcing scheme, SecureInfer adopts an information-theoretic and threat-informed partitioning strategy: security-sensitive components, including non-linear layers, projection of attention head, FNN transformations, and LoRA adapters, are executed inside an SGX enclave, while other linear operations (matrix multiplication) are performed on the GPU after encryption and are securely restored within the enclave. We implement a prototype of SecureInfer using the LLaMA-2 model and evaluate it across performance and security metrics. Our results show that SecureInfer offers strong security guarantees with reasonable performance, offering a practical solution for secure on-device model inference.