# Text Style Transfer for Bias Mitigation using Masked Language Modeling

arXiv: https://arxiv.org/abs/2201.08643 · HTML: https://mlsanity.com/paper/2201.08643/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 21 Jan 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Unsupervised Machine Translation Using Monolingual Corpora Only](https://mlsanity.com/paper/1711.00043/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfer](https://mlsanity.com/paper/1804.06437/) (2018)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Style Transfer Through Back-Translation](https://mlsanity.com/paper/1804.09000/) (2018)
- [Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://mlsanity.com/paper/1908.09368/) (2019)
- [Unsupervised Text Style Transfer using Language Models as Discriminators](https://mlsanity.com/paper/1805.11749/) (2018)
- [Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://mlsanity.com/paper/1805.07685/) (2018)
- ["Mask and Infill" : Applying Masked Language Model to Sentiment Transfer](https://mlsanity.com/paper/1908.08039/) (2019)
- [Personalized Machine Translation: Preserving Original Author Traits](https://mlsanity.com/paper/1610.05461/) (2016)
- [They, Them, Theirs: Rewriting with Gender-Neutral English](https://mlsanity.com/paper/2102.06788/) (2021)
- [Detect and Perturb: Neutral Rewriting of Biased and Sensitive Text via Gradient-based Decoding](https://mlsanity.com/paper/2109.11708/) (2021)
- [Multiple-Attribute Text Style Transfer](https://mlsanity.com/paper/1811.00552/) (2018)
- [Style Transfer as Unsupervised Machine Translation](https://mlsanity.com/paper/1808.07894/) (2018)

## Built on by (most cited first, 28 of 28)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://mlsanity.com/paper/2306.07799/) (2023)
- [Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models](https://mlsanity.com/paper/2501.17420/) (2025)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Walking in Others' Shoes: How Perspective-Taking Guides Large Language Models in Reducing Toxicity and Bias](https://mlsanity.com/paper/2407.15366/) (2024)
- [Queer People are People First: Deconstructing Sexual Identity Stereotypes in Large Language Models](https://mlsanity.com/paper/2307.00101/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Potential and Challenges of Model Editing for Social Debiasing](https://mlsanity.com/paper/2402.13462/) (2024)
- [On the Impact of Noise in Differentially Private Text Rewriting](https://mlsanity.com/paper/2501.19022/) (2025)
- [JAMDEC: Unsupervised Authorship Obfuscation using Constrained Decoding over Small Language Models](https://mlsanity.com/paper/2402.08761/) (2024)
- [Fairness in Large Language Models in Three Hours](https://mlsanity.com/paper/2408.00992/) (2024)
- [Translating away Translationese without Parallel Data](https://mlsanity.com/paper/2310.18830/) (2023)
- [Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing](https://mlsanity.com/paper/2503.15815/) (2025)
- [A Group Fairness Lens for Large Language Models](https://mlsanity.com/paper/2312.15478/) (2023)
- [SC2: Towards Enhancing Content Preservation and Style Consistency in Long Text Style Transfer](https://mlsanity.com/paper/2406.04578/) (2024)
- [FairFlow: An Automated Approach to Model-based Counterfactual Data Augmentation For NLP](https://mlsanity.com/paper/2407.16431/) (2024)
- [Chain-of-Thought Driven Adversarial Scenario Extrapolation for Robust Language Models](https://mlsanity.com/paper/2505.17089/) (2025)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Mitigating Bias in Text Classification via Prompt-Based Text Transformation](https://mlsanity.com/paper/2305.06166/) (2023)
- [Model-based Counterfactual Generator for Gender Bias Mitigation](https://mlsanity.com/paper/2311.03186/) (2023)
- [IDT: Dual-Task Adversarial Attacks for Privacy Protection](https://mlsanity.com/paper/2406.19642/) (2024)
- [Alleviating Choice Supportive Bias in LLM with Reasoning Dependency Generation](https://mlsanity.com/paper/2512.03082/) (2025)
- [A Representation-Level Assessment of Bias Mitigation in Foundation Models](https://mlsanity.com/paper/2604.08561/) (2026)
- [When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation](https://mlsanity.com/paper/2607.07937/) (2026)
- [PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization](https://mlsanity.com/paper/2608.29624/) (2026)

## Abstract (arXiv)

It is well known that textual data on the internet and other digital platforms contain significant levels of bias and stereotypes. Although many such texts contain stereotypes and biases that inherently exist in natural language for reasons that are not necessarily malicious, there are crucial reasons to mitigate these biases. For one, these texts are being used as training corpus to train language models for salient applications like cv-screening, search engines, and chatbots; such applications are turning out to produce discriminatory results. Also, several research findings have concluded that biased texts have significant effects on the target demographic groups. For instance, masculine-worded job advertisements tend to be less appealing to female applicants. In this paper, we present a text style transfer model that can be used to automatically debias textual data. Our style transfer model improves on the limitations of many existing style transfer techniques such as loss of content information. Our model solves such issues by combining latent content encoding with explicit keyword replacement. We will show that this technique produces better content preservation whilst maintaining good style transfer accuracy.