# MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

arXiv: https://arxiv.org/abs/2604.18914 · HTML: https://mlsanity.com/paper/2604.18914/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus](https://mlsanity.com/paper/2006.05754/) (2020)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation](https://mlsanity.com/paper/2211.01355/) (2022)
- [MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models](https://mlsanity.com/paper/2410.17578/) (2024)
- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages?](https://mlsanity.com/paper/1910.14161/) (2019)
- [The Arabic Parallel Gender Corpus 2.0: Extensions and Analyses](https://mlsanity.com/paper/2110.09216/) (2021)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs](https://mlsanity.com/paper/2407.18786/) (2024)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [Attributing Culture-Conditioned Generations to Pretraining Corpora](https://mlsanity.com/paper/2412.20760/) (2024)
- [Advancements in Scientific Controllable Text Generation Methods](https://mlsanity.com/paper/2307.05538/) (2023)
- [A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applications](https://mlsanity.com/paper/2503.20302/) (2025)
- [What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages](https://mlsanity.com/paper/2407.09704/) (2024)
- [Large Language Models and Arabic Content: A Review](https://mlsanity.com/paper/2505.08004/) (2025)
- [Enhancing Scientific Visual Question Answering via Vision-Caption aware Supervised Fine-Tuning](https://mlsanity.com/paper/2509.16628/) (2025)
- [Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning](https://mlsanity.com/paper/2406.00022/) (2024)
- [Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning](https://mlsanity.com/paper/2406.08931/) (2024)

## Abstract (arXiv)

While multilingual large language models (LLMs) perform well on high-level tasks like translation and question answering, their ability to handle grammatical gender and morphological agreement remains underexplored. In morphologically rich languages, gender influences verb conjugation, pronouns, and even first-person constructions with explicit and implicit mentions of gender. We introduce MORPHOGEN, a morphologically grounded large-scale benchmark dataset for evaluating gender-aware generation in three typologically diverse grammatically gendered languages: French, Arabic, and Hindi. The core task, GENFORM, requires models to rewrite a first-person sentence in the opposite gender while preserving its meaning and structure. We construct a high-quality synthetic dataset spanning these three languages and benchmark 15 popular multilingual LLMs (2B-70B) on their ability to perform this transformation. Our results reveal significant gaps and interesting insights into how current models handle morphological gender. MORPHOGEN provides a focused diagnostic lens for gender-aware language modeling and lays the groundwork for future research on inclusive and morphology-sensitive NLP.