ProText: A benchmark dataset for measuring (mis)gendering in long-form texts
Not on the map. Why?
Abstract (arXiv)
We introduce ProText, a dataset for measuring gendering and misgendering in stylistically diverse long-form English texts. ProText spans three dimensions: Theme nouns (names, occupations, titles, kinship terms), Theme category (stereotypically male, stereotypically female, gender-neutral/non-gendered), and Pronoun category (masculine, feminine, gender-neutral, none). The dataset is designed to probe (mis)gendering in text transformations such as summarization and rewrites using state-of-the-art Large Language Models, extending beyond traditional pronoun resolution benchmarks and beyond the gender binary. We validated ProText through a mini case study, showing that even with just two prompts and two models, we can draw nuanced insights regarding gender bias, stereotyping, misgendering, and gendering. We reveal systematic gender bias, particularly when inputs contain no explicit gender cues or when models default to heteronormative assumptions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 47 references that are in the map. First posted on arXiv 29 Mar 2026.
Builds on
31 of 47 references.
References · 31
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
- 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
- 2018Gender Bias in Coreference Resolutioncited by 442
- 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
- 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
- 2019On Measuring Social Biases in Sentence Encoderscited by 398
- 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
- 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
19 more
- 2019Measuring Bias in Contextualized Word Representationscited by 276
- 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
- 2019Evaluating Gender Bias in Machine Translationcited by 249
- 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
- 2018Gender Bias in Neural Natural Language Processingcited by 211
- 2019Gender Bias in Contextualized Word Embeddingscited by 203
- 2023Gender bias and stereotypes in Large Language Modelscited by 187
- 2018Learning Gender-Neutral Word Embeddingscited by 169
- 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
- 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
- 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
- 2019Getting Gender Right in Neural Machine Translationcited by 98
- 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
- 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
- 2023Nationality Bias in Text Generationcited by 50
- 2024Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflictscited by 15
- 2024Locating and Mitigating Gender Bias in Large Language Modelscited by 12
- 2025A Multilingual, Culture-First Approach to Addressing Misgendering in LLM Applicationscited by 6
- 2024Protected group bias and stereotypes in Large Language Modelscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.