# The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

arXiv: https://arxiv.org/abs/2609.09048 · HTML: https://mlsanity.com/paper/2609.09048/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [Large Language Models Often Know When They Are Being Evaluated](https://mlsanity.com/paper/2505.23836/) (2025)
- [What's in a Name? Auditing Large Language Models for Race and Gender Bias](https://mlsanity.com/paper/2402.14875/) (2024)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- ["You Gotta be a Doctor, Lin": An Investigation of Name-Based Bias of Large Language Models in Employment Recommendations](https://mlsanity.com/paper/2406.12232/) (2024)
- [The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring](https://mlsanity.com/paper/2405.04412/) (2024)
- [Measuring Bias or Measuring the Task: Understanding the Brittle Nature of LLM Gender Biases](https://mlsanity.com/paper/2509.04373/) (2025)
- [Can LLMs Hire Fairly? Racial Bias in Resume Screening](https://mlsanity.com/paper/2606.28978/) (2026)
- [FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation](https://mlsanity.com/paper/2607.28934/) (2026)
- [BiAxisBias: Evaluating LLM Bias Beyond a Single Prompt and a Single Explanation](https://mlsanity.com/paper/2605.09041/) (2026)

## Abstract (arXiv)

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.