# Assessing Bias in Metric Models for LLM Open-Ended Generation Bias Benchmarks

arXiv: https://arxiv.org/abs/2410.11059 · HTML: https://mlsanity.com/paper/2410.11059/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Social Chemistry 101: Learning to Reason about Social and Moral Norms](https://mlsanity.com/paper/2011.00620/) (2020)
- [Towards Auditing Large Language Models: Improving Text-based Stereotype Detection](https://mlsanity.com/paper/2311.14126/) (2023)
- [SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration](https://mlsanity.com/paper/2409.11149/) (2024)

## Built on by (most cited first, 4 of 4)

- [Real-World Gaps in AI Governance Research](https://mlsanity.com/paper/2505.00174/) (2025)
- [Social Bias in Popular Question-Answering Benchmarks](https://mlsanity.com/paper/2505.15553/) (2025)
- [Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues](https://mlsanity.com/paper/2601.12868/) (2026)
- [The Benchmark Trap: Structures of Power and Injustice in AI Evaluations](https://mlsanity.com/paper/2608.15326/) (2026)

## Abstract (arXiv)

Open-generation bias benchmarks evaluate social biases in Large Language Models (LLMs) by analyzing their outputs. However, the classifiers used in analysis often have inherent biases, leading to unfair conclusions. This study examines such biases in open-generation benchmarks like BOLD and SAGED. Using the MGSD dataset, we conduct two experiments. The first uses counterfactuals to measure prediction variations across demographic groups by altering stereotype-related prefixes. The second applies explainability tools (SHAP) to validate that the observed biases stem from these counterfactuals. Results reveal unequal treatment of demographic descriptors, calling for more robust bias metric models.