# GPU-based Private Information Retrieval for On-Device Machine Learning Inference

arXiv: https://arxiv.org/abs/2301.10904 · HTML: https://mlsanity.com/paper/2301.10904/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 108 references that are in the map.
- First posted on arXiv 26 Jan 2023.

## Builds on (most cited first, 30 of 32)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Gazelle: A Low Latency Framework for Secure Neural Network Inference](https://mlsanity.com/paper/1801.05507/) (2018)
- [CrypTen: Secure Multi-Party Computation Meets Machine Learning](https://mlsanity.com/paper/2109.00984/) (2021)
- [The Architectural Implications of Facebook's DNN-based Personalized Recommendation](https://mlsanity.com/paper/1906.03109/) (2019)
- [Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models](https://mlsanity.com/paper/2104.05158/) (2021)
- [FALCON: Honest-Majority Maliciously Secure Framework for Private Deep Learning](https://mlsanity.com/paper/2004.02229/) (2020)
- [CrypTFlow: Secure TensorFlow Inference](https://mlsanity.com/paper/1909.07814/) (2019)
- [Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems](https://mlsanity.com/paper/2003.05622/) (2020)
- [DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference](https://mlsanity.com/paper/2001.02772/) (2020)
- [CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU](https://mlsanity.com/paper/2104.10949/) (2021)
- [EdgeRec: Recommender System on Edge in Mobile Taobao](https://mlsanity.com/paper/2005.08416/) (2020)
- [F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption (Extended Version)](https://mlsanity.com/paper/2109.05371/) (2021)
- [On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation](https://mlsanity.com/paper/2204.11091/) (2022)
- [ARIANN: Low-Interaction Privacy-Preserving Deep Learning via Function Secret Sharing](https://mlsanity.com/paper/2006.04593/) (2020)
- [BTS: An Accelerator for Bootstrappable Fully Homomorphic Encryption](https://mlsanity.com/paper/2112.15479/) (2021)
- [HEAX: An Architecture for Computing on Encrypted Data](https://mlsanity.com/paper/1909.09731/) (2019)
- [GuardNN: Secure Accelerator Architecture for Privacy-Preserving Deep Learning](https://mlsanity.com/paper/2008.11632/) (2020)
- [TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU](https://mlsanity.com/paper/2212.14191/) (2022)
- [MGX: Near-Zero Overhead Memory Protection for Data-Intensive Accelerators](https://mlsanity.com/paper/2004.09679/) (2020)
- [FAB: An FPGA-based Accelerator for Bootstrappable Fully Homomorphic Encryption](https://mlsanity.com/paper/2207.11872/) (2022)
- [Riposte: An Anonymous Messaging System Handling Millions of Users](https://mlsanity.com/paper/1503.06115/) (2015)
- [LAORAM: A Look Ahead ORAM Architecture for Training Large Embedding Tables](https://mlsanity.com/paper/2107.08094/) (2021)
- [FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learning](https://mlsanity.com/paper/2206.03852/) (2022)
- [Fast keyed hash/pseudo-random function using SIMD multiply and permute](https://mlsanity.com/paper/1612.06257/) (2016)
- [EHAP-ORAM: Efficient Hardware-Assisted Persistent ORAM System for Non-volatile Memory](https://mlsanity.com/paper/2011.03669/) (2020)

## Built on by (most cited first, 5 of 5)

- [MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation](https://mlsanity.com/paper/2407.01972/) (2024)
- [TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval](https://mlsanity.com/paper/2502.20969/) (2025)
- [A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems](https://mlsanity.com/paper/2406.17335/) (2024)
- [Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inference](https://mlsanity.com/paper/2309.04875/) (2023)
- [CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation](https://mlsanity.com/paper/2511.19711/) (2025)

## Abstract (arXiv)

On-device machine learning (ML) inference can enable the use of private user data on user devices without revealing them to remote servers. However, a pure on-device solution to private ML inference is impractical for many applications that rely on embedding tables that are too large to be stored on-device. In particular, recommendation models typically use multiple embedding tables each on the order of 1-10 GBs of data, making them impractical to store on-device. To overcome this barrier, we propose the use of private information retrieval (PIR) to efficiently and privately retrieve embeddings from servers without sharing any private information. As off-the-shelf PIR algorithms are usually too computationally intensive to directly use for latency-sensitive inference tasks, we 1) propose novel GPU-based acceleration of PIR, and 2) co-design PIR with the downstream ML application to obtain further speedup. Our GPU acceleration strategy improves system throughput by more than 20 × over an optimized CPU PIR implementation, and our PIR-ML co-design provides an over 5 × additional throughput improvement at fixed model quality. Together, for various on-device ML applications such as recommendation and language modeling, our system on a single V100 GPU can serve up to 100,000 queries per second -- a >100 × throughput improvement over a CPU-based baseline -- while maintaining model accuracy.