# Evaluating Gender Bias in Hindi-English Machine Translation arXiv: https://arxiv.org/abs/2106.08680 ยท HTML: https://mlsanity.com/paper/2106.08680/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Builds on 18 of its 34 references that are in the map. - First posted on arXiv 16 Jun 2021. ## Builds on (most cited first, 18 of 18) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016) - [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020) - [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016) - [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019) - [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019) - [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017) - [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019) - [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018) - [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019) - [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021) - [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017) - [Attenuating Bias in Word Vectors](https://mlsanity.com/paper/1901.07656/) (2019) - [On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://mlsanity.com/paper/1905.11684/) (2019) - [Understanding Undesirable Word Embedding Associations](https://mlsanity.com/paper/1908.06361/) (2019) - [Examining Gender Bias in Languages with Grammatical Gender](https://mlsanity.com/paper/1909.02224/) (2019) - [OpenNMT: Neural Machine Translation Toolkit](https://mlsanity.com/paper/1805.11462/) (2018) - [How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages?](https://mlsanity.com/paper/1910.14161/) (2019) ## Built on by (most cited first, 19 of 19) - [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021) - [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022) - [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022) - [Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task](https://mlsanity.com/paper/2306.15298/) (2023) - [What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns](https://mlsanity.com/paper/2305.16051/) (2023) - [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023) - [What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study](https://mlsanity.com/paper/2410.00545/) (2024) - [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023) - [Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias](https://mlsanity.com/paper/2204.10365/) (2022) - [Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation](https://mlsanity.com/paper/2505.08546/) (2025) - [Mitigating Gender Stereotypes in Hindi and Marathi](https://mlsanity.com/paper/2205.05901/) (2022) - [Gender Inflected or Bias Inflicted: On Using Grammatical Gender Cues for Bias Evaluation in Machine Translation](https://mlsanity.com/paper/2311.03767/) (2023) - [Efficient Gender Debiasing of Pre-trained Indic Language Models](https://mlsanity.com/paper/2209.03661/) (2022) - [Uncertainty Quantification for Evaluating Machine Translation Bias](https://mlsanity.com/paper/2507.18338/) (2025) - [Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context](https://mlsanity.com/paper/2603.07792/) (2026) - [Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architectures](https://mlsanity.com/paper/2603.17952/) (2026) - [Investigating Cross-Linguistic Gender Bias in Hindi-English Across Domains](https://mlsanity.com/paper/2111.11159/) (2021) - [Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation](https://mlsanity.com/paper/2312.03710/) (2023) - [Cultural Fidelity in English-to-Hindi Translation: A Preservation-Fluency Frontier for Gender Recoverability](https://mlsanity.com/paper/2605.27654/) (2026) ## Abstract (arXiv) With language models being deployed increasingly in the real world, it is essential to address the issue of the fairness of their outputs. The word embedding representations of these language models often implicitly draw unwanted associations that form a social bias within the model. The nature of gendered languages like Hindi, poses an additional problem to the quantification and mitigation of bias, owing to the change in the form of the words in the sentence, based on the gender of the subject. Additionally, there is sparse work done in the realm of measuring and debiasing systems for Indic languages. In our work, we attempt to evaluate and quantify the gender bias within a Hindi-English machine translation system. We implement a modified version of the existing TGBI metric based on the grammatical considerations for Hindi. We also compare and contrast the resulting bias measurements across multiple metrics for pre-trained embeddings and the ones learned by our machine translation model.