mlsanityData to 14 Oct 2026

Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling

arXiv 2311.18711 ↗First posted 30 Nov 2023cs.CLcited by 12

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present GEST -- a new manually created dataset designed to measure gender-stereotypical reasoning in language models and machine translation systems. GEST contains samples for 16 gender stereotypes about men and women (e.g., Women are beautiful, Men are leaders) that are compatible with the English language and 9 Slavic languages. The definition of said stereotypes was informed by gender experts. We used GEST to evaluate English and Slavic masked LMs, English generative LMs, and machine translation systems. We discovered significant and consistent amounts of gender-stereotypical reasoning in almost all the evaluated models and languages. Our experiments confirm the previously postulated hypothesis that the larger the model, the more stereotypical it usually is.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 46 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

24 of 46 references.

References · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  3. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  4. 2019Evaluating Gender Bias in Machine Translationcited by 249
  5. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  6. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  7. 2021Gender Bias in Machine Translationcited by 117
  8. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  9. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  10. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  11. 2021Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processingcited by 68
  12. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
12 more
  1. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 60
  2. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  3. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  4. 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
  5. 2022Choose Your Lenses: Flaws in Gender Bias Evaluationcited by 29
  6. 2021Evaluating Gender Bias in Hindi-English Machine Translationcited by 19
  7. 2023GATE: A Challenge Set for Gender-Ambiguous Translation Examplescited by 18
  8. 2021Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Modelscited by 15
  9. 2021Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectivescited by 14
  10. 2021How Reliable are Model Diagnostics?cited by 13
  11. 2023In-Depth Look at Word Filling Societal Bias Measurescited by 8
  12. 2022Undesirable Biases in NLP: Addressing Challenges of Measurementcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  2. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  3. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  4. 2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
  5. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3
  6. 2025Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contextscited by 2
  7. 2026Harsher on Male? Evaluating LLMs on Gender-Asymmetric Moral Framing Across Diverse Conflict Scenarioscited by 2
  8. 2025Identifying Gender Stereotypes and Biases in Automated Translation from English to Italian using Similarity Networkscited by 1
  9. 2025Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theorycited by 1
  10. 2025A Detailed Study on LLM Biases Concerning Corporate Social Responsibility and Green Supply Chainscited by 0
  11. 2026MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generationcited by 0
  12. 2026Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Informationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Translate With Care: Addressing Gender Bias, Neutrality, and Reasoning in Large Language Model Translations
  2. 2025An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation
  3. 2025Probing Gender Bias in Multilingual LLMs: A Case Study of Stereotypes in Persian
  4. 2023Gender bias and stereotypes in Large Language Models
  5. 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models
  6. 2025Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lens
  7. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
  8. 2025DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoning