# Beyond Memorization: Violating Privacy Via Inference with Large Language Models

arXiv: https://arxiv.org/abs/2310.07298 · HTML: https://mlsanity.com/paper/2310.07298/

## Key facts

- Cited by 175 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 62 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [ProPILE: Probing Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2307.01881/) (2023)

## Built on by (most cited first, 30 of 175)

- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Large Language Model Unlearning via Embedding-Corrupted Prompts](https://mlsanity.com/paper/2406.07933/) (2024)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration](https://mlsanity.com/paper/2311.06062/) (2023)
- [AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents](https://mlsanity.com/paper/2503.09780/) (2025)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [A Closer Look at Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2410.08109/) (2024)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [Can LLMs Follow Simple Rules?](https://mlsanity.com/paper/2311.04235/) (2023)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [SoK: Memorization in General-Purpose Large Language Models](https://mlsanity.com/paper/2310.18362/) (2023)
- [Private Attribute Inference from Images with Vision-Language Models](https://mlsanity.com/paper/2404.10618/) (2024)
- [Offset Unlearning for Large Language Models](https://mlsanity.com/paper/2404.11045/) (2024)
- [A Synthetic Dataset for Personal Attribute Inference](https://mlsanity.com/paper/2406.07217/) (2024)
- [Protecting Your LLMs with Information Bottleneck](https://mlsanity.com/paper/2404.13968/) (2024)
- [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024)
- [From Assistant to Double Agent: Formalizing and Benchmarking Attacks on OpenClaw for Personalized Local AI Agent](https://mlsanity.com/paper/2602.08412/) (2026)
- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act](https://mlsanity.com/paper/2410.07959/) (2024)
- [IncogniText: Privacy-enhancing Conditional Text Anonymization via LLM-based Private Attribute Randomization](https://mlsanity.com/paper/2407.02956/) (2024)
- [Large Language Models are Advanced Anonymizers](https://mlsanity.com/paper/2402.13846/) (2024)
- [Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models](https://mlsanity.com/paper/2407.13934/) (2024)
- [Can Language Models Recognize Convincing Arguments?](https://mlsanity.com/paper/2404.00750/) (2024)
- [A Survey on Privacy Risks and Protection in Large Language Models](https://mlsanity.com/paper/2505.01976/) (2025)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [Robustly Improving LLM Fairness in Realistic Settings via Interpretability](https://mlsanity.com/paper/2506.10922/) (2025)

## Abstract (arXiv)

Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models' inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals' privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to 85% top-1 and 95% top-3 accuracy at a fraction of the cost (100×) and time (240×) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.