mlsanityData to 7 Oct 2026

This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models

arXiv 2305.12757 ↗First posted 22 May 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Bias research in NLP seeks to analyse models for social biases, thus helping NLP practitioners uncover, measure, and mitigate social harms. We analyse the body of work that uses prompts and templates to assess bias in language models. We draw on a measurement modelling framework to create a taxonomy of attributes that capture what a bias test aims to measure and how that measurement is carried out. By applying this taxonomy to 90 bias tests, we illustrate qualitatively and quantitatively that core aspects of bias test conceptualisations and operationalisations are frequently unstated or ambiguous, carry implicit assumptions, or be mismatched. Our analysis illuminates the scope of possible bias types the field is able to measure, and reveals types that are as yet under-researched. We offer guidance to enable the community to explore a wider section of the possible bias space, and to better close the gap between desired outcomes and experimental design, both for bias and for evaluating language models more broadly.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

32 of 51 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  6. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  7. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  10. 2019Measurement and Fairnesscited by 299
  11. 2019Measuring Bias in Contextualized Word Representationscited by 275
  12. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
20 more
  1. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  2. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  3. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  4. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  5. 2019Towards a Critical Race Methodology in Algorithmic Fairnesscited by 124
  6. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  7. 2021Mitigating Language-Dependent Ethnic Bias in BERTcited by 87
  8. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  9. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  10. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  11. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  12. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 52
  13. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  14. 2022Quantifying Social Biases Using Templates is Unreliablecited by 38
  15. 2021Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lenscited by 30
  16. 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
  17. 2022Challenges in Measuring Bias via Open-Ended Language Generationcited by 23
  18. 2022Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddingscited by 18
  19. 2022PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Modelcited by 9
  20. 2021Ensemble of MRR and NDCG models for Visual Dialogcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 4.

9 papers.

  1. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  2. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  3. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  4. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  5. 2025Agree to Disagree? A Meta-Evaluation of LLM Misgenderingcited by 6
  6. 2023Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Modelscited by 3
  7. 2025BiasFreeBench: a Benchmark for Mitigating Bias in Large Language Model Responsescited by 2
  8. 2025Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commercecited by 1
  9. 2025EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answeringcited by 1