# PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior

arXiv: https://arxiv.org/abs/2605.12147 · HTML: https://mlsanity.com/paper/2605.12147/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 20 of 20)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [Questioning the Survey Responses of Large Language Models](https://mlsanity.com/paper/2306.07951/) (2023)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Towards Automating Data Access Permissions in AI Agents](https://mlsanity.com/paper/2511.17959/) (2025)
- [The Need for a Socially-Grounded Persona Framework for User Simulation](https://mlsanity.com/paper/2601.07110/) (2026)
- [Discovering Smart Home Internet of Things Privacy Norms Using Contextual Integrity](https://mlsanity.com/paper/1805.06031/) (2018)
- [The Feasibility of Dynamically Granted Permissions: Aligning Mobile Privacy with User Preferences](https://mlsanity.com/paper/1703.02090/) (2017)
- [Improving Language Model Personas via Rationalization with Psychological Scaffolds](https://mlsanity.com/paper/2504.17993/) (2025)
- [Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective](https://mlsanity.com/paper/2508.06760/) (2025)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?](https://mlsanity.com/paper/2602.18464/) (2026)
- [Text-Based Personas for Simulating User Privacy Decisions](https://mlsanity.com/paper/2603.19791/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor because they often diverge from the user's actual privacy behavior. (3) Users with high AI/chatbot experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions. We release PrivacySIM to enable further evaluation of LLM privacy simulation.