mlsanityData to 14 Oct 2026

AgentFairBench: Do LLM Agents Discriminate When They Act?

arXiv 2606.16723 ↗First posted 15 Jun 2026cs.AIcited by 3

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers. We introduce AgentFairBench, a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents. Grounded in a companion framework, the Bias Conduction Framework (BCF, restated here), it spans three regulator-anchored domains: hiring, lending, and medical triage. Synthetic, demographic-neutral profiles are evaluated in counterfactual matched sets that vary only a name-coded race x gender signal (in the Bertrand Mullainathan tradition), under four agent scaffolds of increasing agency (direct, chain-of-thought, multi-agent deliberation, tool-augmented). A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity, with bootstrap confidence intervals, paired tests, and false-discovery-rate control, for single-digit dollars per model. A live leaderboard with a held-out private split and a contamination canary admits external models by submission. Our pilot (864 decisions plus a test-retest replication) carries a methodological lesson: comparing a six-group score spread against a two-run noise difference overstates disparity by ~ 2.4X through statistic arity alone. Against an arity matched noise floor and an omnibus group test, claude haiku 4 5 shows no demographic effect above sampling noise (0 of 120 pairwise and 0 of 9 omnibus contrasts survive correction); a planted-bias test confirms the instrument detects disparity when present. The contribution is a sound, sensitive, adoption-ready instrument, the arity matched null methodology, and open artifacts to scale it. Code, data, and harness are released under open licenses, with an anonymized review artifact.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 29 of its 57 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

29 of 57 references.

References · 29

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  8. 2017Counterfactual Fairnesscited by 804
  9. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  11. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
17 more
  1. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  2. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  3. 2025Survey on Evaluation of LLM-based Agentscited by 204
  4. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  5. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  6. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  7. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
  8. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
  9. 2024What's in a Name? Auditing Large Language Models for Race and Gender Biascited by 51
  10. 2025Robustly Improving LLM Fairness in Realistic Settings via Interpretabilitycited by 18
  11. 2025FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answeringcited by 6
  12. 2026Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMscited by 6
  13. 2025Emergent Bias and Fairness in Multi-Agent Decision Systemscited by 5
  14. 2026Compared to What? Baselines and Metrics for Counterfactual Promptingcited by 5
  15. 2026EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triagecited by 3
  16. 2025mFARM: Towards Multi-Faceted Fairness Assessment based on HARMs in Clinical Decision Supportcited by 2
  17. 2026Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance Systemcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgentcited by 0
  2. 2026Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runscited by 0
  3. 2026RelLedger: Relational Information Governance for LLM Agent Organizationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MALIBU Benchmark: Multi-Agent LLM Implicit Bias Uncovered
  2. 2025Evaluating the Bias in LLMs for Surveying Opinion and Decision Making in Healthcare
  3. 2026First, Do No Harm (With LLMs): Mitigating Racial Bias via Agentic Workflows
  4. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  5. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  6. 2025FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks
  7. 2025EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Law
  8. 2024Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions