# Evaluating the Hidden Costs of Personalization in Large Language Models

arXiv: https://arxiv.org/abs/2608.28833 · HTML: https://mlsanity.com/paper/2608.28833/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 28 Aug 2026.

## Builds on (most cited first, 30 of 30)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants](https://mlsanity.com/paper/2506.09902/) (2025)
- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://mlsanity.com/paper/2506.00137/) (2025)
- [Interaction Context Often Increases Sycophancy in LLMs](https://mlsanity.com/paper/2509.12517/) (2025)
- [PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation](https://mlsanity.com/paper/2503.01303/) (2025)
- [CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions](https://mlsanity.com/paper/2508.01674/) (2025)
- [Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach](https://mlsanity.com/paper/2505.18882/) (2025)
- [Personalize Your LLM: Fake it then Align it](https://mlsanity.com/paper/2503.01048/) (2025)
- [PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization](https://mlsanity.com/paper/2506.12915/) (2025)
- [Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking](https://mlsanity.com/paper/2402.05880/) (2024)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [Diversity-Enhanced Reasoning for Subjective Questions](https://mlsanity.com/paper/2507.20187/) (2025)
- [How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants](https://mlsanity.com/paper/2601.16621/) (2026)
- [Privacy in Human-AI Romantic Relationships: Concerns, Boundaries, and Agency](https://mlsanity.com/paper/2601.16824/) (2026)
- [A Conditional Companion: Lived Experiences of People with Mental Health Disorders Using LLMs](https://mlsanity.com/paper/2602.00402/) (2026)
- [When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information](https://mlsanity.com/paper/2605.10013/) (2026)

## Abstract (arXiv)

While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, and user profiles. Specifically, we identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use of personalization in AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLM personalization and how personalized information shapes its responses. Our empirical analysis across 13 LLMs demonstrates the presence of user profiles and retrieved memories consistently exacerbates biases, resulting in an average drop of 45.9% in irrelevant personalization, 41.7% in preference narrowing and 61.7% in sycophantic bias.