# Locating and Controlling Implicit Personalization in Large Language Models

arXiv: https://arxiv.org/abs/2608.11735 · HTML: https://mlsanity.com/paper/2608.11735/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 29 of 29)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [Learning without training: The implicit dynamics of in-context learning](https://mlsanity.com/paper/2507.16003/) (2025)
- [Who's asking? User personas and the mechanics of latent misalignment](https://mlsanity.com/paper/2406.12094/) (2024)
- [FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering](https://mlsanity.com/paper/2504.14492/) (2025)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- [Refusal in LLMs is an Affine Function](https://mlsanity.com/paper/2411.09003/) (2024)
- [Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization](https://mlsanity.com/paper/2505.16467/) (2025)
- [Implicit Personalization in Language Models: A Systematic Study](https://mlsanity.com/paper/2405.14808/) (2024)
- [Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing](https://mlsanity.com/paper/2501.14457/) (2025)
- [Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers](https://mlsanity.com/paper/2606.02776/) (2026)
- [Fair outputs, Biased Internals: Causal Potency and Asymmetry of Latent Bias in LLMs for High-Stakes Decisions](https://mlsanity.com/paper/2605.15217/) (2026)

## Built on by (most cited first, 2 of 2)

- [User Model Extraction via Belief Self-Distillation](https://mlsanity.com/paper/2609.31603/) (2026)
- [MIRROR: From Imitation to Internalization in LLM Personalization](https://mlsanity.com/paper/2610.09795/) (2026)

## Abstract (arXiv)

Large language models (LLMs) often shift their outputs in response to implicit demographic cues even when users never state a demographic identity. Previous work has documented this behavior, but the connection between these behavioral changes and the model's internal activations remains unclear. Using matched cued and neutral conversations across five LLMs, we establish that a localized internal activation signal tracks changes in recommendations, with correlations up to r=0.87. When multiple cues appear together, their internal signals largely combine, but the changes in output do not simply add up. We further show that removing the internal signal associated with one cue can suppress its influence, often more effectively than asking the model to ignore demographics via prompting, while largely preserving general benchmark performance. However, the ability to selectively remove one dimension's influence while leaving co-present dimensions intact remains highly model- and attribute-specific. These results connect implicit personalization behavior to an internal signal that can be analyzed and causally controlled.