# BiAxisBias: Evaluating LLM Bias Beyond a Single Prompt and a Single Explanation

arXiv: https://arxiv.org/abs/2605.09041 · HTML: https://mlsanity.com/paper/2605.09041/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 9 May 2026.

## Builds on (most cited first, 11 of 11)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- [Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge](https://mlsanity.com/paper/2504.07887/) (2025)
- [Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency](https://mlsanity.com/paper/2402.15481/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits](https://mlsanity.com/paper/2609.09048/) (2026)

## Abstract (arXiv)

LLM bias scores can depend on audit design. We introduce BiAxisBias, a prespecified audit varying task, role, perspective, sentiment, and wording over 200 stereotype statements while retaining forced Selection and Rationale as separate protocol readouts. Its main matrix spans eight LLMs and 401 templates (641,600 responses). Across five equivalent questions, 17.1% of 1,600 model-statement pairs change Selection. With three observations per unit in both arms, instability averages 10.5% across all ten three-wording subsets, versus 6.3% for three identical calls. Across four controlled task paradigms, 9/28 model pairs reverse; a seven-model factorial sensitivity identifies task-by-sentiment as the largest two-way component (raw eta-squared = 0.0465). In 10,000 equal-budget resampling draws, mean absolute error against a declared 18-condition finite reference is 10.63 points for one-template concentration, 1.86 for matrix-wide simple random sampling, and 1.75 for condition stratification; ranking inversions are 20.6%, 4.8%, and 5.5%. Thus broad coverage drives the gain, while stratification has only a modest score-error advantage and no ranking advantage over random sampling. In a separate forced-output diagnostic, task-specific Selection mappings and judge-coded Rationale stance disagree in 34.2% of 7,959 dual-valid responses (31.2% versus 3.0% by direction). This diagnoses output-contract sensitivity, not two validated measures of one construct.