# OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings

arXiv: https://arxiv.org/abs/2007.00049 · HTML: https://mlsanity.com/paper/2007.00049/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 30 of 40)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models](https://mlsanity.com/paper/2305.11840/) (2023)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [MISGENDERED: Limits of Large Language Models in Understanding Pronouns](https://mlsanity.com/paper/2306.03950/) (2023)
- [The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks](https://mlsanity.com/paper/2210.10040/) (2022)
- [MABEL: Attenuating Gender Bias using Textual Entailment Data](https://mlsanity.com/paper/2210.14975/) (2022)
- [Socially Aware Bias Measurements for Hindi Language Representations](https://mlsanity.com/paper/2110.07871/) (2021)
- [Language Models Get a Gender Makeover: Mitigating Gender Bias with Few-Shot Data Interventions](https://mlsanity.com/paper/2306.04597/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases](https://mlsanity.com/paper/2307.01595/) (2023)
- [Bias and Fairness in Chatbots: An Overview](https://mlsanity.com/paper/2309.08836/) (2023)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [Learning Fair Representations via Rate-Distortion Maximization](https://mlsanity.com/paper/2202.00035/) (2022)
- [Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency](https://mlsanity.com/paper/2402.15481/) (2024)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Marked Attribute Bias in Natural Language Inference](https://mlsanity.com/paper/2109.14039/) (2021)
- [Erasure of Unaligned Attributes from Neural Representations](https://mlsanity.com/paper/2302.02997/) (2023)
- [Removing Spurious Concepts from Neural Network Representations via Joint Subspace Estimation](https://mlsanity.com/paper/2310.11991/) (2023)
- [Fundamental Limits of Perfect Concept Erasure](https://mlsanity.com/paper/2503.20098/) (2025)
- [On the Mutual Influence of Gender and Occupation in LLM Representations](https://mlsanity.com/paper/2503.06792/) (2025)
- [Preserving Task-Relevant Information Under Linear Concept Removal](https://mlsanity.com/paper/2506.10703/) (2025)
- [VERB: Visualizing and Interpreting Bias Mitigation Techniques for Word Representations](https://mlsanity.com/paper/2104.02797/) (2021)
- [SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)ata](https://mlsanity.com/paper/2311.17259/) (2023)

## Abstract (arXiv)

Language representations are known to carry stereotypical biases and, as a result, lead to biased predictions in downstream tasks. While existing methods are effective at mitigating biases by linear projection, such methods are too aggressive: they not only remove bias, but also erase valuable information from word embeddings. We develop new measures for evaluating specific information retention that demonstrate the tradeoff between bias removal and information retention. To address this challenge, we propose OSCaR (Orthogonal Subspace Correction and Rectification), a bias-mitigating method that focuses on disentangling biased associations between concepts instead of removing concepts wholesale. Our experiments on gender biases show that OSCaR is a well-balanced approach that ensures that semantic information is retained in the embeddings and bias is also effectively mitigated.