# Exploring Safety-Utility Trade-Offs in Personalized Language Models

arXiv: https://arxiv.org/abs/2406.11107 · HTML: https://mlsanity.com/paper/2406.11107/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 75 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 30 of 55)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)

## Built on by (most cited first, 27 of 27)

- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [A Survey on Model Extraction Attacks and Defenses for Large Language Models](https://mlsanity.com/paper/2506.22521/) (2025)
- [Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions](https://mlsanity.com/paper/2512.12775/) (2025)
- [PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI](https://mlsanity.com/paper/2512.24848/) (2025)
- [The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation](https://mlsanity.com/paper/2501.07849/) (2025)
- [Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Models](https://mlsanity.com/paper/2502.15086/) (2025)
- [One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalization](https://mlsanity.com/paper/2601.18572/) (2026)
- [The Power of Personality: A Human Simulation Perspective to Investigate Large Language Model Agents](https://mlsanity.com/paper/2502.20859/) (2025)
- [The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior](https://mlsanity.com/paper/2509.19364/) (2025)
- [The Rise of Darkness: Safety-Utility Trade-Offs in Role-Playing Dialogue Agents](https://mlsanity.com/paper/2502.20757/) (2025)
- [Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering](https://mlsanity.com/paper/2505.15805/) (2025)
- [Evaluating Large Language Models for Detecting Antisemitism](https://mlsanity.com/paper/2509.18293/) (2025)
- [When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs](https://mlsanity.com/paper/2601.11000/) (2026)
- [From Anger to Joy: How Nationality Personas Shape Emotion Attribution in Large Language Models](https://mlsanity.com/paper/2506.02431/) (2025)
- [Mind the Gap! Pathways Towards Unifying AI Safety and Ethics Research](https://mlsanity.com/paper/2512.10058/) (2025)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [OpenSafeIntent: Evaluating Intent-Calibrated Safe Completion Across Dual-Use Prompt Sets](https://mlsanity.com/paper/2607.02047/) (2026)
- [Identity-Robust Language Model Generation via Content Integrity Preservation](https://mlsanity.com/paper/2601.09141/) (2026)
- [The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?](https://mlsanity.com/paper/2501.13952/) (2025)
- [Debiasing LLMs by Masking Unfairness-Driving Attention Heads](https://mlsanity.com/paper/2510.10142/) (2025)
- [Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education](https://mlsanity.com/paper/2601.14506/) (2026)
- [Large Language Models in Peer-Run Community Behavioral Health Services: Understanding Peer Specialists and Service Users' Perspectives on Opportunities, Risks, and Mitigation Strategies](https://mlsanity.com/paper/2602.08187/) (2026)
- [UGID: Unified Graph Isomorphism for Debiasing Large Language Models](https://mlsanity.com/paper/2603.19144/) (2026)
- [Mitigating LLM biases toward spurious social contexts using direct preference optimization](https://mlsanity.com/paper/2604.02585/) (2026)
- [TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignment](https://mlsanity.com/paper/2606.01755/) (2026)
- [SHARD: Safe and Helpful Alignment via Self-Reframing Distillation](https://mlsanity.com/paper/2606.15517/) (2026)
- [Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languages](https://mlsanity.com/paper/2608.22490/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become increasingly integrated into daily applications, it is essential to ensure they operate fairly across diverse user demographics. In this work, we show that LLMs suffer from personalization bias, where their performance is impacted when they are personalized to a user's identity. We quantify personalization bias by evaluating the performance of LLMs along two axes - safety and utility. We measure safety by examining how benign LLM responses are to unsafe prompts with and without personalization. We measure utility by evaluating the LLM's performance on various tasks, including general knowledge, mathematical abilities, programming, and reasoning skills. We find that various LLMs, ranging from open-source models like Llama (Touvron et al., 2023) and Mistral (Jiang et al., 2023) to API-based ones like GPT-3.5 and GPT-4o (Ouyang et al., 2022), exhibit significant variance in performance in terms of safety-utility trade-offs depending on the user's identity. Finally, we discuss several strategies to mitigate personalization bias using preference tuning and prompt-based defenses.