# Personalized Language Model Learning on Text Data Without User Identifiers

arXiv: https://arxiv.org/abs/2501.06062 · HTML: https://mlsanity.com/paper/2501.06062/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 10 Jan 2025.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Adaptive Personalized Federated Learning](https://mlsanity.com/paper/2003.13461/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Product-based Neural Networks for User Response Prediction](https://mlsanity.com/paper/1611.00144/) (2016)
- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Federated Learning of N-gram Language Models](https://mlsanity.com/paper/1910.03432/) (2019)
- [RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation](https://mlsanity.com/paper/2306.07206/) (2023)
- [Explainable Recommendation with Personalized Review Retrieval and Aspect Learning](https://mlsanity.com/paper/2306.12657/) (2023)
- [Exploring and Adapting Chinese GPT to Pinyin Input Method](https://mlsanity.com/paper/2203.00249/) (2022)

## Built on by (most cited first, 3 of 3)

- [Personalized Text Generation with Contrastive Activation Steering](https://mlsanity.com/paper/2503.05213/) (2025)
- [Collaborative Learning of On-Device Small Model and Cloud-Based Large Model: Advances and Future Directions](https://mlsanity.com/paper/2504.15300/) (2025)
- [From Profiles to Steering Vectors: Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation](https://mlsanity.com/paper/2607.21620/) (2026)

## Abstract (arXiv)

In many practical natural language applications, user data are highly sensitive, requiring anonymous uploads of text data from mobile devices to the cloud without user identifiers. However, the absence of user identifiers restricts the ability of cloud-based language models to provide personalized services, which are essential for catering to diverse user needs. The trivial method of replacing an explicit user identifier with a static user embedding as model input still compromises data anonymization. In this work, we propose to let each mobile device maintain a user-specific distribution to dynamically generate user embeddings, thereby breaking the one-to-one mapping between an embedding and a specific user. We further theoretically demonstrate that to prevent the cloud from tracking users via uploaded embeddings, the local distributions of different users should either be derived from a linearly dependent space to avoid identifiability or be close to each other to prevent accurate attribution. Evaluation on both public and industrial datasets using different language models reveals a remarkable improvement in accuracy from incorporating anonymous user embeddings, while preserving real-time inference requirement.