# User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies

arXiv: https://arxiv.org/abs/2509.05382 · HTML: https://mlsanity.com/paper/2509.05382/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Builds on (most cited first, 8 of 8)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [The Foundation Model Transparency Index](https://mlsanity.com/paper/2310.12941/) (2023)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [The 2024 Foundation Model Transparency Index](https://mlsanity.com/paper/2407.12929/) (2024)
- [Machine Learners Should Acknowledge the Legal Implications of Large Language Models as Personal Data](https://mlsanity.com/paper/2503.01630/) (2025)

## Built on by (most cited first, 9 of 9)

- [The 2025 Foundation Model Transparency Index](https://mlsanity.com/paper/2512.10169/) (2025)
- [When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs](https://mlsanity.com/paper/2601.11000/) (2026)
- [A Longitudinal Measurement of Privacy Policy Evolution for Large Language Models](https://mlsanity.com/paper/2511.21758/) (2025)
- [Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU](https://mlsanity.com/paper/2605.20706/) (2026)
- [Black Box Absorption: LLMs Undermining Innovative Ideas](https://mlsanity.com/paper/2510.20612/) (2025)
- [KadiAssistant: A conversational AI Agent for information retrieval in Kadi4Mat](https://mlsanity.com/paper/2605.18850/) (2026)
- [Short paper: Models in the dark -- Rectification and erasure under GDPR in ML supply chains](https://mlsanity.com/paper/2606.05946/) (2026)
- [Assessment of Non-Institutional AI Tool Usage Among Clinicians](https://mlsanity.com/paper/2609.11990/) (2026)
- [SpliTEE: Fast and Private LLM Inference by Coupling GPU-Assisted Trusted Execution Environments with Differential Privacy](https://mlsanity.com/paper/2609.15039/) (2026)

## Abstract (arXiv)

Hundreds of millions of people now regularly interact with large language models via chatbots. Model developers are eager to acquire new sources of high-quality training data as they race to improve model capabilities and win market share. This paper analyzes the privacy policies of six U.S. frontier AI developers to understand how they use their users' chats to train models. Drawing primarily on the California Consumer Privacy Act, we develop a novel qualitative coding schema that we apply to each developer's relevant privacy policies to compare data collection and use practices across the six companies. We find that all six developers appear to employ their users' chat data to train and improve their models by default, and that some retain this data indefinitely. Developers may collect and train on personal information disclosed in chats, including sensitive information such as biometric and health data, as well as files uploaded by users. Four of the six companies we examined appear to include children's chat data for model training, as well as customer data from other products. On the whole, developers' privacy policies often lack essential information about their practices, highlighting the need for greater transparency and accountability. We address the implications of users' lack of consent for the use of their chat data for model training, data security issues arising from indefinite chat data retention, and training on children's chat data. We conclude by providing recommendations to policymakers and developers to address the data privacy challenges posed by LLM-powered chatbots.