# CAPE: Context-Aware Private Embeddings for Private Language Learning

arXiv: https://arxiv.org/abs/2108.12318 · HTML: https://mlsanity.com/paper/2108.12318/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference](https://mlsanity.com/paper/1906.11798/) (2019)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [Embedding Text in Hyperbolic Spaces](https://mlsanity.com/paper/1806.04313/) (2018)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness](https://mlsanity.com/paper/2010.01285/) (2020)
- [Privacy-preserving Neural Representations of Text](https://mlsanity.com/paper/1808.09408/) (2018)
- [Generalised Differential Privacy for Text Document Processing](https://mlsanity.com/paper/1811.10256/) (2018)
- [Towards Demystifying Membership Inference Attacks](https://mlsanity.com/paper/1807.09173/) (2018)
- [Heterogeneous Differential Privacy](https://mlsanity.com/paper/1504.06998/) (2015)
- [Compositional Demographic Word Embeddings](https://mlsanity.com/paper/2010.02986/) (2020)
- [I Am Not What I Write: Privacy Preserving Text Representation Learning](https://mlsanity.com/paper/1907.03189/) (2019)
- [Adversarial Learning of Privacy-Preserving Text Representations for De-Identification of Medical Records](https://mlsanity.com/paper/1906.05000/) (2019)
- [dpUGC: Learn Differentially Private Representation for User Generated Contents](https://mlsanity.com/paper/1903.10453/) (2019)
- [Scalable Differential Privacy with Certified Robustness in Adversarial Learning](https://mlsanity.com/paper/1903.09822/) (2019)

## Built on by (most cited first, 22 of 22)

- [DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Models](https://mlsanity.com/paper/2310.20138/) (2023)
- [Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services](https://mlsanity.com/paper/2305.06212/) (2023)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [Differentially Private Natural Language Models: Recent Advances and Future Directions](https://mlsanity.com/paper/2301.09112/) (2023)
- [A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking The Privacy-Utility Trade-off](https://mlsanity.com/paper/2404.03324/) (2024)
- [The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts](https://mlsanity.com/paper/2406.14318/) (2024)
- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)
- [Fair NLP Models with Differentially Private Text Encoders](https://mlsanity.com/paper/2205.06135/) (2022)
- [Privacy-Preserving Language Model Inference with Instance Obfuscation](https://mlsanity.com/paper/2402.08227/) (2024)
- [Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks](https://mlsanity.com/paper/2504.17421/) (2025)
- [Private Language Models via Truncated Laplacian Mechanism](https://mlsanity.com/paper/2410.08027/) (2024)
- [PrIeD-KIE: Towards Privacy Preserved Document Key Information Extraction](https://mlsanity.com/paper/2310.03777/) (2023)
- [Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection](https://mlsanity.com/paper/2409.15907/) (2024)
- [Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2502.00451/) (2025)
- [Anti-adversarial Learning: Desensitizing Prompts for Large Language Models](https://mlsanity.com/paper/2505.01273/) (2025)
- [Protecting Privacy in Classifiers by Token Manipulation](https://mlsanity.com/paper/2407.01334/) (2024)
- [Investigating User Perspectives on Differentially Private Text Privatization](https://mlsanity.com/paper/2503.09338/) (2025)
- [DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Models](https://mlsanity.com/paper/2508.04208/) (2025)
- [Demystifying the Privacy-Utility Trade-off in LLM Interactions](https://mlsanity.com/paper/2609.10992/) (2026)
- [Not All Entities are Created Equal: A Dynamic Anonymization Framework for Privacy-Preserving RAG](https://mlsanity.com/paper/2603.26074/) (2026)
- [DeepInvert: Semi-Supervised Embedding Inversion Against Obfuscated Language Models](https://mlsanity.com/paper/2608.04477/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)

## Abstract (arXiv)

Deep learning-based language models have achieved state-of-the-art results in a number of applications including sentiment analysis, topic labelling, intent classification and others. Obtaining text representations or embeddings using these models presents the possibility of encoding personally identifiable information learned from language and context cues that may present a risk to reputation or privacy. To ameliorate these issues, we propose Context-Aware Private Embeddings (CAPE), a novel approach which preserves privacy during training of embeddings. To maintain the privacy of text representations, CAPE applies calibrated noise through differential privacy, preserving the encoded semantic links while obscuring sensitive information. In addition, CAPE employs an adversarial training regime that obscures identified private variables. Experimental results demonstrate that the proposed approach reduces private information leakage better than either single intervention.