# Diverse Adversaries for Mitigating Bias in Training

arXiv: https://arxiv.org/abs/2101.10001 · HTML: https://mlsanity.com/paper/2101.10001/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 25 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Towards Robust and Privacy-preserving Text Representations](https://mlsanity.com/paper/1805.06093/) (2018)
- [What's in a Domain? Learning Domain-Robust Text Representations using Adversarial Training](https://mlsanity.com/paper/1805.06088/) (2018)

## Built on by (most cited first, 30 of 48)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering](https://mlsanity.com/paper/2504.14492/) (2025)
- [Balancing out Bias: Achieving Fairness Through Balanced Training](https://mlsanity.com/paper/2109.08253/) (2021)
- [Contrastive Learning for Fair Representations](https://mlsanity.com/paper/2109.10645/) (2021)
- [Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information](https://mlsanity.com/paper/2203.07893/) (2022)
- [MABEL: Attenuating Gender Bias using Textual Entailment Data](https://mlsanity.com/paper/2210.14975/) (2022)
- [Evaluating Debiasing Techniques for Intersectional Biases](https://mlsanity.com/paper/2109.10441/) (2021)
- [SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents](https://mlsanity.com/paper/2411.03284/) (2024)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Parameter-efficient Modularised Bias Mitigation via AdapterFusion](https://mlsanity.com/paper/2302.06321/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Optimising Equal Opportunity Fairness in Model Training](https://mlsanity.com/paper/2205.02393/) (2022)
- [BLIND: Bias Removal With No Demographics](https://mlsanity.com/paper/2212.10563/) (2022)
- [Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases](https://mlsanity.com/paper/2307.01595/) (2023)
- [Fairness-aware Class Imbalanced Learning](https://mlsanity.com/paper/2109.10444/) (2021)
- [Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworks](https://mlsanity.com/paper/2205.15171/) (2022)
- [Large Language Model Bias Mitigation from the Perspective of Knowledge Editing](https://mlsanity.com/paper/2405.09341/) (2024)
- [Co2PT: Mitigating Bias in Pre-trained Language Models through Counterfactual Contrastive Prompt Tuning](https://mlsanity.com/paper/2310.12490/) (2023)
- [Identifying and Mitigating Social Bias Knowledge in Language Models](https://mlsanity.com/paper/2408.11843/) (2024)
- [A Chinese Dataset for Evaluating the Safeguards in Large Language Models](https://mlsanity.com/paper/2402.12193/) (2024)
- [Conditional Supervised Contrastive Learning for Fair Text Classification](https://mlsanity.com/paper/2205.11485/) (2022)
- [Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP](https://mlsanity.com/paper/2302.05711/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Fair NLP Models with Differentially Private Text Encoders](https://mlsanity.com/paper/2205.06135/) (2022)
- [Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions](https://mlsanity.com/paper/2311.07115/) (2023)
- [Towards Equal Opportunity Fairness through Adversarial Learning](https://mlsanity.com/paper/2203.06317/) (2022)
- [FairGrad: Fairness Aware Gradient Descent](https://mlsanity.com/paper/2206.10923/) (2022)
- [Effective Controllable Bias Mitigation for Classification and Retrieval using Gate Adapters](https://mlsanity.com/paper/2401.16457/) (2024)
- [DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision](https://mlsanity.com/paper/2303.08403/) (2023)

## Abstract (arXiv)

Adversarial learning can learn fairer and less biased models of language than standard methods. However, current adversarial techniques only partially mitigate model bias, added to which their training procedures are often unstable. In this paper, we propose a novel approach to adversarial learning based on the use of multiple diverse discriminators, whereby discriminators are encouraged to learn orthogonal hidden representations from one another. Experimental results show that our method substantially improves over standard adversarial removal methods, in terms of reducing bias and the stability of training.