# Driving Context into Text-to-Text Privatization

arXiv: https://arxiv.org/abs/2306.01457 · HTML: https://mlsanity.com/paper/2306.01457/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge](https://mlsanity.com/paper/1908.07245/) (2019)
- [SenseBERT: Driving Some Sense into BERT](https://mlsanity.com/paper/1908.05646/) (2019)
- [Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text](https://mlsanity.com/paper/1910.08917/) (2019)
- [TEM: High Utility Metric Differential Privacy on Text](https://mlsanity.com/paper/2107.07928/) (2021)
- [A Differentially Private Text Perturbation Method Using a Regularized Mahalanobis Metric](https://mlsanity.com/paper/2010.11947/) (2020)
- [Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space](https://mlsanity.com/paper/1504.06654/) (2015)
- [AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes](https://mlsanity.com/paper/1507.01127/) (2015)
- [On a Utilitarian Approach to Privacy Preserving Text Generation](https://mlsanity.com/paper/2104.11838/) (2021)
- [De-Conflated Semantic Representations](https://mlsanity.com/paper/1608.01961/) (2016)
- [A Mixture Model for Learning Multi-Sense Word Embeddings](https://mlsanity.com/paper/1706.05111/) (2017)
- [Making Sense of Word Embeddings](https://mlsanity.com/paper/1708.03390/) (2017)
- [Breaking Sticks and Ambiguities with Adaptive Skip-gram](https://mlsanity.com/paper/1502.07257/) (2015)
- [Multi-sense embeddings through a word sense disambiguation process](https://mlsanity.com/paper/2101.08700/) (2021)
- [MUSE: Modularizing Unsupervised Sense Embeddings](https://mlsanity.com/paper/1704.04601/) (2017)
- [A Simple and Efficient Method To Generate Word Sense Representations](https://mlsanity.com/paper/1412.6045/) (2014)
- [Differentially Private Adversarial Robustness Through Randomized Perturbations](https://mlsanity.com/paper/2009.12718/) (2020)

## Built on by (most cited first, 13 of 13)

- [Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation](https://mlsanity.com/paper/2309.11765/) (2023)
- [Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning](https://mlsanity.com/paper/2410.12085/) (2024)
- [Private Language Models via Truncated Laplacian Mechanism](https://mlsanity.com/paper/2410.08027/) (2024)
- [Inspecting the Representation Manifold of Differentially-Private Text](https://mlsanity.com/paper/2503.14991/) (2025)
- [STAMP: Selective Task-Aware Mechanism for Text Privacy](https://mlsanity.com/paper/2603.12237/) (2026)
- [Investigating User Perspectives on Differentially Private Text Privatization](https://mlsanity.com/paper/2503.09338/) (2025)
- [The Double-edged Sword of LLM-based Data Reconstruction: Understanding and Mitigating Contextual Vulnerability in Word-level Differential Privacy Text Sanitization](https://mlsanity.com/paper/2508.18976/) (2025)
- [A Systematic Exploration of Text Decomposition and Budget Distribution in Differentially Private Text Obfuscation](https://mlsanity.com/paper/2605.01065/) (2026)
- [Characterizing Stereotypical Bias from Privacy-preserving Pre-Training](https://mlsanity.com/paper/2407.00764/) (2024)
- [Stereo-LiDAR Fusion by Semi-Global Matching With Discrete Disparity-Matching Cost and Semidensification](https://mlsanity.com/paper/2504.05148/) (2025)
- [One Size Fits All? A Modular Adaptive Sanitization Kit (MASK) for Customizable Privacy-Preserving Phone Scam Detection](https://mlsanity.com/paper/2510.18493/) (2025)
- [From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop](https://mlsanity.com/paper/2608.11171/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)

## Abstract (arXiv)

textitMetric Differential Privacy enables text-to-text privatization by adding calibrated noise to the vector of a word derived from an embedding space and projecting this noisy vector back to a discrete vocabulary using a nearest neighbor search. Since words are substituted without context, this mechanism is expected to fall short at finding substitutes for words with ambiguous meanings, such as textit'bank'. To account for these ambiguous words, we leverage a sense embedding and incorporate a sense disambiguation step prior to noise injection. We encompass our modification to the privatization mechanism with an estimation of privacy and utility. For word sense disambiguation on the textitWords in Context dataset, we demonstrate a substantial increase in classification accuracy by 6.05%.