# This Prompt is Measuring : Evaluating Bias Evaluation in Language Models arXiv: https://arxiv.org/abs/2305.12757 ยท HTML: https://mlsanity.com/paper/2305.12757/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 51 references that are in the map. - First posted on arXiv 22 May 2023. ## Builds on (most cited first, 30 of 32) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020) - [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020) - [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020) - [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019) - [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021) - [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019) - [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019) - [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018) - [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019) - [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019) - [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020) - [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021) - [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020) - [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019) - [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021) - [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021) - [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020) - [On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations](https://mlsanity.com/paper/2203.13928/) (2022) - [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022) - [Investigating African-American Vernacular English in Transformer-Based Text Generation](https://mlsanity.com/paper/2010.02510/) (2020) - [Investigating Societal Biases in a Poetry Composition System](https://mlsanity.com/paper/2011.02686/) (2020) - [Quantifying Social Biases Using Templates is Unreliable](https://mlsanity.com/paper/2210.04337/) (2022) - [Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lens](https://mlsanity.com/paper/2110.00521/) (2021) - [Challenges in Measuring Bias via Open-Ended Language Generation](https://mlsanity.com/paper/2205.11601/) (2022) - [Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models](https://mlsanity.com/paper/2109.03300/) (2021) - [Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings](https://mlsanity.com/paper/2203.07523/) (2022) ## Built on by (most cited first, 9 of 9) - [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023) - [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024) - [Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models](https://mlsanity.com/paper/2310.07589/) (2023) - [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024) - [Agree to Disagree? A Meta-Evaluation of LLM Misgendering](https://mlsanity.com/paper/2504.17075/) (2025) - [Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models](https://mlsanity.com/paper/2308.12578/) (2023) - [BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responses](https://mlsanity.com/paper/2510.00232/) (2025) - [Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commerce](https://mlsanity.com/paper/2503.13518/) (2025) - [EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering](https://mlsanity.com/paper/2507.11216/) (2025) ## Abstract (arXiv) Bias research in NLP seeks to analyse models for social biases, thus helping NLP practitioners uncover, measure, and mitigate social harms. We analyse the body of work that uses prompts and templates to assess bias in language models. We draw on a measurement modelling framework to create a taxonomy of attributes that capture what a bias test aims to measure and how that measurement is carried out. By applying this taxonomy to 90 bias tests, we illustrate qualitatively and quantitatively that core aspects of bias test conceptualisations and operationalisations are frequently unstated or ambiguous, carry implicit assumptions, or be mismatched. Our analysis illuminates the scope of possible bias types the field is able to measure, and reveals types that are as yet under-researched. We offer guidance to enable the community to explore a wider section of the possible bias space, and to better close the gap between desired outcomes and experimental design, both for bias and for evaluating language models more broadly.