# Improving User Privacy in Personalized Generation: Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculations

arXiv: https://arxiv.org/abs/2601.17569 · HTML: https://mlsanity.com/paper/2601.17569/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 35 of its 72 references that are in the map.
- First posted on arXiv 24 Jan 2026.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 30 of 35)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data](https://mlsanity.com/paper/2406.14773/) (2024)
- [LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordination](https://mlsanity.com/paper/2309.11696/) (2023)
- [Personalized Generation In Large Model Era: A Survey](https://mlsanity.com/paper/2503.02614/) (2025)
- [Learning to Rewrite Prompts for Personalized Text Generation](https://mlsanity.com/paper/2310.00152/) (2023)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [User Modeling and User Profiling: A Comprehensive Survey](https://mlsanity.com/paper/2402.09660/) (2024)
- [Reasoning-Enhanced Self-Training for Long-Form Personalized Text Generation](https://mlsanity.com/paper/2501.04167/) (2025)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generation](https://mlsanity.com/paper/2501.14956/) (2025)
- [Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models](https://mlsanity.com/paper/2409.09510/) (2024)
- [Measuring Personalization of Web Search](https://mlsanity.com/paper/1706.05011/) (2017)
- [SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation](https://mlsanity.com/paper/2506.12699/) (2025)
- [Corporate Communication Companion (CCC): An LLM-empowered Writing Assistant for Workplace Social Media](https://mlsanity.com/paper/2405.04656/) (2024)
- [Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximization](https://mlsanity.com/paper/2410.09942/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Personalized Language Models via Privacy-Preserving Evolutionary Model Merging](https://mlsanity.com/paper/2503.18008/) (2025)
- [Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection](https://mlsanity.com/paper/2504.13938/) (2025)

## Built on by (most cited first, 1 of 1)

- [PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generation](https://mlsanity.com/paper/2610.14713/) (2026)

## Abstract (arXiv)

Personalization is crucial for aligning Large Language Model (LLM) outputs with individual user preferences and background knowledge. State-of-the-art solutions are based on retrieval augmentation, where relevant context from a user profile is retrieved for LLM consumption. These methods deal with a trade-off between exposing retrieved private data to cloud providers and relying on less capable local models. We introduce P3, an interactive framework for high-quality personalization without revealing private profiles to server-side LLMs. In P3, a large server-side model generates a sequence of k draft tokens based solely on the user query, while a small client-side model, with retrieval access to the user's private profile, evaluates and modifies these drafts to better reflect user preferences. This process repeats until an end token is generated. Experiments on LaMP-QA, a recent benchmark consisting of three personalized question answering datasets, show that P3 consistently outperforms both non-personalized server-side and personalized client-side baselines, achieving statistically significant improvements of 7.4% to 9% on average. Importantly, P3 recovers 90.3% to 95.7% of the utility of a ``leaky'' upper-bound scenario in which the full profile is exposed to the large server-side model. Privacy analyses, including linkability and attribute inference attacks, indicate that P3 preserves the privacy of a non-personalized server-side model, introducing only marginal additional leakage (1.5%--3.5%) compared to submitting a query without any personal context. Additionally, the framework is efficient for edge deployment, with the client-side model generating only 9.2% of the total tokens. These results demonstrate that P3 provides a practical, effective solution for personalized generation with improved privacy.