# Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts

arXiv: https://arxiv.org/abs/2606.30152 · HTML: https://mlsanity.com/paper/2606.30152/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023)
- [Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models](https://mlsanity.com/paper/2101.09688/) (2021)
- [Examining Gender Bias in Languages with Grammatical Gender](https://mlsanity.com/paper/1909.02224/) (2019)
- [OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings](https://mlsanity.com/paper/2007.00049/) (2020)
- [Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations](https://mlsanity.com/paper/2401.05792/) (2024)
- [How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages?](https://mlsanity.com/paper/1910.14161/) (2019)
- [Grammatical gender associations outweigh topical gender bias in crosslinguistic word embeddings](https://mlsanity.com/paper/2005.08864/) (2020)
- [Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Models](https://mlsanity.com/paper/2511.18123/) (2025)
- [Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals](https://mlsanity.com/paper/2206.01691/) (2022)
- [Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora](https://mlsanity.com/paper/2406.13677/) (2024)
- [What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages](https://mlsanity.com/paper/2407.09704/) (2024)
- [EuroGEST: Investigating gender stereotypes in multilingual language models](https://mlsanity.com/paper/2506.03867/) (2025)
- [Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models](https://mlsanity.com/paper/2508.03199/) (2025)
- [The Causal Influence of Grammatical Gender on Distributional Semantics](https://mlsanity.com/paper/2311.18567/) (2023)
- [What is "Typological Diversity" in NLP?](https://mlsanity.com/paper/2402.04222/) (2024)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [Mitigating Gender Bias in Contextual Word Embeddings](https://mlsanity.com/paper/2411.12074/) (2024)

## Abstract (arXiv)

Contextual language models conflate grammatical gender and social semantic bias in gendered languages such as Spanish. Existing gender debiasing approaches only operate on static word embeddings leaving contextual representations unexplored for this two dimensional gender disentanglement. To address the this issue, we make the first attempt to disentangle grammatical gender from semantic contamination for contextual embeddings. We construct both controlled templates and natural Wikipedia contexts to build balanced datasets of inanimate nouns, and design a framework equipped with centroid, Support Vector Machine (SVM) and Linear Discriminant Analysis (LDA) gender direction estimators as well as contamination-aware weighting strategies. A set of dual-objective evaluation metrics is proposed to balance the suppression of grammatical gender leakage on inanimate nouns and the preservation of semantic gender distinctions for occupation terms. The results reveal that unweighted controlled contexts yield the purest grammatical gender direction, and the centroid estimator achieves better performance than discriminative baselines.