# You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personas

arXiv: https://arxiv.org/abs/2205.10228 · HTML: https://mlsanity.com/paper/2205.10228/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 26 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- [Learning Controllable Fair Representations](https://mlsanity.com/paper/1812.04218/) (2018)
- [DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition](https://mlsanity.com/paper/2012.08695/) (2020)
- [Overlearning Reveals Sensitive Attributes](https://mlsanity.com/paper/1905.11742/) (2019)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [Privacy-preserving Neural Representations of Text](https://mlsanity.com/paper/1808.09408/) (2018)
- [Privacy Regularization: Joint Privacy-Utility Optimization in Language Models](https://mlsanity.com/paper/2103.07567/) (2021)
- [OTTers: One-turn Topic Transitions for Open-Domain Dialogue](https://mlsanity.com/paper/2105.13710/) (2021)
- [Getting To Know You: User Attribute Extraction from Dialogues](https://mlsanity.com/paper/1908.04621/) (2019)
- [Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs](https://mlsanity.com/paper/2105.12049/) (2021)
- [TIPRDC: Task-Independent Privacy-Respecting Data Crowdsourcing Framework for Deep Learning with Anonymized Intermediate Representations](https://mlsanity.com/paper/2005.11480/) (2020)
- [Listening between the Lines: Learning Personal Attributes from Conversations](https://mlsanity.com/paper/1904.10887/) (2019)
- [Federated Deep Learning with Bayesian Privacy](https://mlsanity.com/paper/2109.13012/) (2021)

## Built on by (most cited first, 11 of 11)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects](https://mlsanity.com/paper/2305.02750/) (2023)
- [Privacy in Large Language Models: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2310.10383/) (2023)
- [PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://mlsanity.com/paper/2406.01394/) (2024)
- [Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2405.17974/) (2024)
- [Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory](https://mlsanity.com/paper/2408.10053/) (2024)
- [Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements](https://mlsanity.com/paper/2302.09270/) (2023)
- [No Free Lunch Theorem for Privacy-Preserving LLM Inference](https://mlsanity.com/paper/2405.20681/) (2024)
- [Through the Lens of Core Competency: Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2308.07902/) (2023)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)

## Abstract (arXiv)

Social chatbots, also known as chit-chat chatbots, evolve rapidly with large pretrained language models. Despite the huge progress, privacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. On the other hand, the datasets used for training chatbots contain many private conversations between two individuals. In this work, we further investigate the privacy leakage of the hidden states of chatbots trained by language modeling which has not been well studied yet. We show that speakers' personas can be inferred through a simple neural network with high accuracy. To this end, we propose effective defense objectives to protect persona leakage from hidden states. We conduct extensive experiments to demonstrate that our proposed defense objectives can greatly reduce the attack accuracy from 37.6% to 0.5%. Meanwhile, the proposed objectives preserve language models' powerful generation ability.