# A Synthetic Dataset for Personal Attribute Inference

arXiv: https://arxiv.org/abs/2406.07217 · HTML: https://mlsanity.com/paper/2406.07217/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour](https://mlsanity.com/paper/2311.09410/) (2023)
- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translation](https://mlsanity.com/paper/1711.01921/) (2017)
- [Private Attribute Inference from Images with Vision-Language Models](https://mlsanity.com/paper/2404.10618/) (2024)
- [Large Language Models are Advanced Anonymizers](https://mlsanity.com/paper/2402.13846/) (2024)
- [AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays](https://mlsanity.com/paper/2304.14276/) (2023)
- [Generating Synthetic Text Data to Evaluate Causal Inference Methods](https://mlsanity.com/paper/2102.05638/) (2021)

## Built on by (most cited first, 30 of 36)

- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [Textual Unlearning Gives a False Sense of Unlearning](https://mlsanity.com/paper/2406.13348/) (2024)
- [Automated Profile Inference with Language Model Agents](https://mlsanity.com/paper/2505.12402/) (2025)
- [A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas](https://mlsanity.com/paper/2505.07850/) (2025)
- [Ads that Talk Back: Implications and Perceptions of Injecting Personalized Advertising into LLM Chatbots](https://mlsanity.com/paper/2409.15436/) (2024)
- [PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs](https://mlsanity.com/paper/2505.12238/) (2025)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Self-Refining Language Model Anonymizers via Adversarial Distillation](https://mlsanity.com/paper/2506.01420/) (2025)
- [Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch](https://mlsanity.com/paper/2602.03183/) (2026)
- [Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs](https://mlsanity.com/paper/2602.11528/) (2026)
- [Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference](https://mlsanity.com/paper/2509.12152/) (2025)
- [Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation](https://mlsanity.com/paper/2604.07486/) (2026)
- [Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM](https://mlsanity.com/paper/2604.17827/) (2026)
- [IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data](https://mlsanity.com/paper/2506.02449/) (2025)
- [PETLP: A Privacy-by-Design Pipeline for Social Media Data in AI Research](https://mlsanity.com/paper/2508.09232/) (2025)
- [Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2509.24488/) (2025)
- [Adaptive Text Anonymization: Learning Privacy-Utility Trade-offs via Prompt Optimization](https://mlsanity.com/paper/2602.20743/) (2026)
- [RedacBench: Can AI Erase Your Secrets?](https://mlsanity.com/paper/2603.20208/) (2026)
- [Can LLM Agents Sustain Long-Horizon Organizational Dynamics?](https://mlsanity.com/paper/2606.01199/) (2026)
- [GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization](https://mlsanity.com/paper/2608.06526/) (2026)
- [Retrieval-Confused Generation is a Good Defender for Privacy Violation Attack of Large Language Models](https://mlsanity.com/paper/2506.19889/) (2025)
- [RL-Finetuned LLMs for Privacy-Preserving Synthetic Rewriting](https://mlsanity.com/paper/2508.19286/) (2025)
- [The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques](https://mlsanity.com/paper/2603.08207/) (2026)
- [Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents](https://mlsanity.com/paper/2605.06232/) (2026)
- [MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions](https://mlsanity.com/paper/2606.14715/) (2026)
- [Attribute Inference from Interactive Targeted Ads](https://mlsanity.com/paper/2606.15209/) (2026)
- [ProfileFoundry: A Synthetic Person-Object Substrate for Privacy, Memory, and Tool-Use Evaluation in LLM Agent](https://mlsanity.com/paper/2606.26403/) (2026)
- [CARD: Controlled Agentic Reddit Discussions for Credit Card Simulation](https://mlsanity.com/paper/2608.09790/) (2026)

## Abstract (arXiv)

Recently, powerful Large Language Models (LLMs) have become easily accessible to hundreds of millions of users world-wide. However, their strong capabilities and vast world knowledge do not come without associated privacy risks. In this work, we focus on the emerging privacy threat LLMs pose -- the ability to accurately infer personal information from online texts. Despite the growing importance of LLM-based author profiling, research in this area has been hampered by a lack of suitable public datasets, largely due to ethical and privacy concerns associated with real personal data. We take two steps to address this problem: (i) we construct a simulation framework for the popular social media platform Reddit using LLM agents seeded with synthetic personal profiles; (ii) using this framework, we generate SynthPAI, a diverse synthetic dataset of over 7800 comments manually labeled for personal attributes. We validate our dataset with a human study showing that humans barely outperform random guessing on the task of distinguishing our synthetic comments from real ones. Further, we verify that our dataset enables meaningful personal attribute inference research by showing across 18 state-of-the-art LLMs that our synthetic comments allow us to draw the same conclusions as real-world data. Combined, our experimental results, dataset and pipeline form a strong basis for future privacy-preserving research geared towards understanding and mitigating inference-based privacy threats that LLMs pose.