# Challenges in Measuring Bias via Open-Ended Language Generation

arXiv: https://arxiv.org/abs/2205.11601 · HTML: https://mlsanity.com/paper/2205.11601/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Challenges in Detoxifying Language Models](https://mlsanity.com/paper/2109.07445/) (2021)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [They, Them, Theirs: Rewriting with Gender-Neutral English](https://mlsanity.com/paper/2102.06788/) (2021)
- [Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand](https://mlsanity.com/paper/2112.04139/) (2021)
- [Neural-based Natural Language Generation in Dialogue using RNN Encoder-Decoder with Semantic Aggregation](https://mlsanity.com/paper/1706.06714/) (2017)

## Built on by (most cited first, 24 of 24)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Evaluation of African American Language Bias in Natural Language Generation](https://mlsanity.com/paper/2305.14291/) (2023)
- [Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization](https://mlsanity.com/paper/2210.04492/) (2022)
- [Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models](https://mlsanity.com/paper/2408.03907/) (2024)
- [Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias](https://mlsanity.com/paper/2212.10678/) (2022)
- [ECBD: Evidence-Centered Benchmark Design for NLP](https://mlsanity.com/paper/2406.08723/) (2024)
- [DUnE: Dataset for Unified Editing](https://mlsanity.com/paper/2311.16087/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models](https://mlsanity.com/paper/2305.12757/) (2023)
- [In-Depth Look at Word Filling Societal Bias Measures](https://mlsanity.com/paper/2302.12640/) (2023)
- [KidLM: Advancing Language Models for Children -- Early Insights and Future Directions](https://mlsanity.com/paper/2410.03884/) (2024)
- [An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation](https://mlsanity.com/paper/2210.03826/) (2022)
- [Agree to Disagree? A Meta-Evaluation of LLM Misgendering](https://mlsanity.com/paper/2504.17075/) (2025)
- [Generative AI in Writing Research Papers: A New Type of Algorithmic Bias and Uncertainty in Scholarly Work](https://mlsanity.com/paper/2312.10057/) (2023)
- [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022)
- [Sports and Women's Sports: Gender Bias in Text Generation with Olympic Data](https://mlsanity.com/paper/2502.04218/) (2025)
- [Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language](https://mlsanity.com/paper/2507.16557/) (2025)
- [AI Evaluation Should Require Standardized Item-Level Data Releases](https://mlsanity.com/paper/2604.03244/) (2026)
- [When LLMs Imagine People: A Human-Centered Persona Brainstorm Audit for Bias and Fairness in Creative Applications](https://mlsanity.com/paper/2602.00044/) (2026)
- [GG-BBQ: German Gender Bias Benchmark for Question Answering](https://mlsanity.com/paper/2507.16410/) (2025)
- [Towards Resource Efficient and Interpretable Bias Mitigation in Large Language Models](https://mlsanity.com/paper/2412.01711/) (2024)
- [Developing A Framework to Support Human Evaluation of Bias in Generated Free Response Text](https://mlsanity.com/paper/2505.03053/) (2025)
- [GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models](https://mlsanity.com/paper/2605.28848/) (2026)

## Abstract (arXiv)

Researchers have devised numerous ways to quantify social biases vested in pretrained language models. As some language models are capable of generating coherent completions given a set of textual prompts, several prompting datasets have been proposed to measure biases between social groups -- posing language generation as a way of identifying biases. In this opinion paper, we analyze how specific choices of prompt sets, metrics, automatic tools and sampling strategies affect bias results. We find out that the practice of measuring biases through text completion is prone to yielding contradicting results under different experiment settings. We additionally provide recommendations for reporting biases in open-ended language generation for a more complete outlook of biases exhibited by a given language model. Code to reproduce the results is released under https://github.com/feyzaakyurek/bias-textgen.