# Towards Robust and Privacy-preserving Text Representations

arXiv: https://arxiv.org/abs/1805.06093 · HTML: https://mlsanity.com/paper/1805.06093/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2018.

## Built on by (most cited first, 30 of 75)

- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization](https://mlsanity.com/paper/2202.00443/) (2022)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Differential Privacy for Text Analytics via Natural Text Sanitization](https://mlsanity.com/paper/2106.01221/) (2021)
- [Natural Language Understanding with Privacy-Preserving BERT](https://mlsanity.com/paper/2104.07504/) (2021)
- [Diverse Adversaries for Mitigating Bias in Training](https://mlsanity.com/paper/2101.10001/) (2021)
- [NeuTral Rewriter: A Rule-Based and Neural Approach to Automatic Rewriting into Gender-Neutral Alternatives](https://mlsanity.com/paper/2109.06105/) (2021)
- [Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks](https://mlsanity.com/paper/2202.08011/) (2022)
- [Balancing out Bias: Achieving Fairness Through Balanced Training](https://mlsanity.com/paper/2109.08253/) (2021)
- [Contrastive Learning for Fair Representations](https://mlsanity.com/paper/2109.10645/) (2021)
- [Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information](https://mlsanity.com/paper/2203.07893/) (2022)
- [Submix: Practical Private Prediction for Large-Scale Language Models](https://mlsanity.com/paper/2201.00971/) (2022)
- [Privacy Regularization: Joint Privacy-Utility Optimization in Language Models](https://mlsanity.com/paper/2103.07567/) (2021)
- [Evaluating Debiasing Techniques for Intersectional Biases](https://mlsanity.com/paper/2109.10441/) (2021)
- [Choose Your Lenses: Flaws in Gender Bias Evaluation](https://mlsanity.com/paper/2210.11471/) (2022)
- [Benchmarking Differential Privacy and Federated Learning for BERT Models](https://mlsanity.com/paper/2106.13973/) (2021)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [Dictionary-based Debiasing of Pre-trained Word Embeddings](https://mlsanity.com/paper/2101.09525/) (2021)
- [CAPE: Context-Aware Private Embeddings for Private Language Learning](https://mlsanity.com/paper/2108.12318/) (2021)
- [Optimising Equal Opportunity Fairness in Model Training](https://mlsanity.com/paper/2205.02393/) (2022)
- [Privacy-Preserving Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2102.09604/) (2021)
- [Fairness-aware Class Imbalanced Learning](https://mlsanity.com/paper/2109.10444/) (2021)
- [How to keep text private? A systematic review of deep learning methods for privacy-preserving natural language processing](https://mlsanity.com/paper/2205.10095/) (2022)
- [Generative Adversarial Networks: A Survey Towards Private and Secure Applications](https://mlsanity.com/paper/2106.03785/) (2021)
- [Learning Fair Representations via Rate-Distortion Maximization](https://mlsanity.com/paper/2202.00035/) (2022)
- [Adversarial Scrubbing of Demographic Information for Text Classification](https://mlsanity.com/paper/2109.08613/) (2021)
- [Adversarial Stylometry in the Wild: Transferable Lexical Substitution Attacks on Author Profiling](https://mlsanity.com/paper/2101.11310/) (2021)

## Abstract (arXiv)

Written text often provides sufficient clues to identify the author, their gender, age, and other important attributes. Consequently, the authorship of training and evaluation corpora can have unforeseen impacts, including differing model performance for different user groups, as well as privacy implications. In this paper, we propose an approach to explicitly obscure important author characteristics at training time, such that representations learned are invariant to these attributes. Evaluating on two tasks, we show that this leads to increased privacy in the learned representations, as well as more robust models to varying evaluation conditions, including out-of-domain corpora.