# Using Natural Sentences for Understanding Biases in Language Models

arXiv: https://arxiv.org/abs/2205.06303 · HTML: https://mlsanity.com/paper/2205.06303/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 6 of 6)

- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)

## Built on by (most cited first, 19 of 19)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks](https://mlsanity.com/paper/2306.05550/) (2023)
- [MISGENDERED: Limits of Large Language Models in Understanding Pronouns](https://mlsanity.com/paper/2306.03950/) (2023)
- [Geographic and Geopolitical Biases of Language Models](https://mlsanity.com/paper/2212.10408/) (2022)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- [Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?](https://mlsanity.com/paper/2404.03134/) (2024)
- [Fairness in Language Models Beyond English: Gaps and Challenges](https://mlsanity.com/paper/2302.12578/) (2023)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges](https://mlsanity.com/paper/2310.11252/) (2023)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages](https://mlsanity.com/paper/2504.18560/) (2025)
- [Auditing Preferences for Brands and Cultures in LLMs](https://mlsanity.com/paper/2603.18300/) (2026)
- [Regional Bias in Large Language Models](https://mlsanity.com/paper/2601.16349/) (2026)
- [Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Models](https://mlsanity.com/paper/2211.14639/) (2022)
- [Template-Based Probes Are Imperfect Lenses for Counterfactual Bias Evaluation in LLMs](https://mlsanity.com/paper/2404.03471/) (2024)
- [Multilingual Text Representation](https://mlsanity.com/paper/2309.00949/) (2023)
- [Exposing the Unsaid: Visualizing Hidden LLM Bias through Stochastic Path Aggregation](https://mlsanity.com/paper/2606.19344/) (2026)

## Abstract (arXiv)

Evaluation of biases in language models is often limited to synthetically generated datasets. This dependence traces back to the need for a prompt-style dataset to trigger specific behaviors of language models. In this paper, we address this gap by creating a prompt dataset with respect to occupations collected from real-world natural sentences present in Wikipedia. We aim to understand the differences between using template-based prompts and natural sentence prompts when studying gender-occupation biases in language models. We find bias evaluations are very sensitive to the design choices of template prompts, and we propose using natural sentence prompts for systematic evaluations to step away from design choices that could introduce bias in the observations.