# Quantifying Social Biases Using Templates is Unreliable

arXiv: https://arxiv.org/abs/2210.04337 · HTML: https://mlsanity.com/paper/2210.04337/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 9 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Reducing Gender Bias in Abusive Language Detection](https://mlsanity.com/paper/1808.07231/) (2018)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Perturbation Sensitivity Analysis to Detect Unintended Model Biases](https://mlsanity.com/paper/1910.04210/) (2019)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Automatic Construction of Evaluation Suites for Natural Language Generation Datasets](https://mlsanity.com/paper/2106.09069/) (2021)
- [On the Impact of Random Seeds on the Fairness of Clinical Classifiers](https://mlsanity.com/paper/2104.06338/) (2021)

## Built on by (most cited first, 30 of 39)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [MISGENDERED: Limits of Large Language Models in Understanding Pronouns](https://mlsanity.com/paper/2306.03950/) (2023)
- [The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations](https://mlsanity.com/paper/2308.02053/) (2023)
- [The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks](https://mlsanity.com/paper/2210.10040/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection](https://mlsanity.com/paper/2501.02295/) (2025)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models](https://mlsanity.com/paper/2406.15484/) (2024)
- [Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?](https://mlsanity.com/paper/2404.03134/) (2024)
- [Perceptions of Linguistic Uncertainty by Language Models and Humans](https://mlsanity.com/paper/2407.15814/) (2024)
- [Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)](https://mlsanity.com/paper/2410.19314/) (2024)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.23398/) (2025)
- [How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion](https://mlsanity.com/paper/2508.03712/) (2025)
- [Are Models Biased on Text without Gender-related Language?](https://mlsanity.com/paper/2405.00588/) (2024)
- [This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models](https://mlsanity.com/paper/2305.12757/) (2023)
- [GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models](https://mlsanity.com/paper/2408.12494/) (2024)
- [WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case](https://mlsanity.com/paper/2409.05653/) (2024)
- [BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models](https://mlsanity.com/paper/2302.07371/) (2023)
- [Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languages](https://mlsanity.com/paper/2305.11673/) (2023)
- [Trustworthy Social Bias Measurement](https://mlsanity.com/paper/2212.11672/) (2022)
- [Agree to Disagree? A Meta-Evaluation of LLM Misgendering](https://mlsanity.com/paper/2504.17075/) (2025)
- ["Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models](https://mlsanity.com/paper/2310.08780/) (2023)
- [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022)
- [Benchmarking Large Language Model Volatility](https://mlsanity.com/paper/2311.15180/) (2023)
- [Mind vs. Mouth: On Measuring Re-judge Inconsistency of Social Bias in Large Language Models](https://mlsanity.com/paper/2308.12578/) (2023)
- [Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems](https://mlsanity.com/paper/2506.04482/) (2025)
- [Failure of contextual invariance in large language models](https://mlsanity.com/paper/2603.23485/) (2026)
- [VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech](https://mlsanity.com/paper/2604.17248/) (2026)

## Abstract (arXiv)

Recently, there has been an increase in efforts to understand how large language models (LLMs) propagate and amplify social biases. Several works have utilized templates for fairness evaluation, which allow researchers to quantify social biases in the absence of test sets with protected attribute labels. While template evaluation can be a convenient and helpful diagnostic tool to understand model deficiencies, it often uses a simplistic and limited set of templates. In this paper, we study whether bias measurements are sensitive to the choice of templates used for benchmarking. Specifically, we investigate the instability of bias measurements by manually modifying templates proposed in previous works in a semantically-preserving manner and measuring bias across these modifications. We find that bias values and resulting conclusions vary considerably across template modifications on four tasks, ranging from an 81% reduction (NLI) to a 162% increase (MLM) in (task-specific) bias measurements. Our results indicate that quantifying fairness in LLMs, as done in current practice, can be brittle and needs to be approached with more care and caution.