# Analyzing Gender Representation in Multilingual Models

arXiv: https://arxiv.org/abs/2204.09168 · HTML: https://mlsanity.com/paper/2204.09168/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [On the Pitfalls of Analyzing Individual Neurons in Language Models](https://mlsanity.com/paper/2110.07483/) (2021)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/1909.06775/) (2019)
- [Examining Gender Bias in Languages with Grammatical Gender](https://mlsanity.com/paper/1909.02224/) (2019)
- [It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT](https://mlsanity.com/paper/2010.08275/) (2020)
- [How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages?](https://mlsanity.com/paper/1910.14161/) (2019)
- [Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi](https://mlsanity.com/paper/2005.00318/) (2020)
- [On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs](https://mlsanity.com/paper/2005.01204/) (2020)
- [Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languages](https://mlsanity.com/paper/2107.10181/) (2021)

## Built on by (most cited first, 10 of 10)

- [Naturalistic Causal Probing for Morpho-Syntax](https://mlsanity.com/paper/2205.07043/) (2022)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [Don't Touch My Diacritics](https://mlsanity.com/paper/2410.24140/) (2024)
- [Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Models](https://mlsanity.com/paper/2403.12809/) (2024)
- [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022)
- [Investigating grammatical abstraction in language models using few-shot learning of novel noun gender](https://mlsanity.com/paper/2403.10338/) (2024)
- [Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation](https://mlsanity.com/paper/2501.10150/) (2025)
- [MultiContrievers: Analysis of Dense Retrieval Representations](https://mlsanity.com/paper/2402.15925/) (2024)
- [Iterative Multilingual Spectral Attribute Erasure](https://mlsanity.com/paper/2506.11244/) (2025)
- [Benchmarking Gender and Political Bias in Large Language Models](https://mlsanity.com/paper/2509.06164/) (2025)

## Abstract (arXiv)

Multilingual language models were shown to allow for nontrivial transfer across scripts and languages. In this work, we study the structure of the internal representations that enable this transfer. We focus on the representation of gender distinctions as a practical case study, and examine the extent to which the gender concept is encoded in shared subspaces across different languages. Our analysis shows that gender representations consist of several prominent components that are shared across languages, alongside language-specific components. The existence of language-independent and language-specific components provides an explanation for an intriguing empirical observation we make: while gender classification transfers well across languages, interventions for gender removal, trained on a single language, do not transfer easily to others.