# Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairness

arXiv: https://arxiv.org/abs/2602.02932 · HTML: https://mlsanity.com/paper/2602.02932/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023)
- [Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?](https://mlsanity.com/paper/2405.16908/) (2024)
- [Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications](https://mlsanity.com/paper/2310.14607/) (2023)
- [LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs](https://mlsanity.com/paper/2406.05194/) (2024)
- [Implicit Bias in LLMs: A Survey](https://mlsanity.com/paper/2503.02776/) (2025)
- [Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth](https://mlsanity.com/paper/2502.20758/) (2025)
- [Enhancing Answer Reliability Through Inter-Model Consensus of Large Language Models](https://mlsanity.com/paper/2411.16797/) (2024)
- [AutoPK: Leveraging LLMs and a Hybrid Similarity Metric for Advanced Retrieval of Pharmacokinetic Data from Complex Tables and Documents](https://mlsanity.com/paper/2510.00039/) (2025)
- [LoopBench: Discovering Emergent Symmetry Breaking Strategies with LLM Swarms](https://mlsanity.com/paper/2512.13713/) (2025)
- [Toward Systematic Counterfactual Fairness Evaluation of Large Language Models: The CAFFE Framework](https://mlsanity.com/paper/2512.16816/) (2025)

## Built on by (most cited first, 4 of 4)

- [Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers](https://mlsanity.com/paper/2606.02776/) (2026)
- [Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment](https://mlsanity.com/paper/2606.01456/) (2026)
- [Same Game, Different Story: A Minimal Conservative Strategic Robustness Benchmark for Large Language Model Agents](https://mlsanity.com/paper/2607.19670/) (2026)
- [Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups](https://mlsanity.com/paper/2605.08671/) (2026)

## Abstract (arXiv)

Prior work on fairness in large language models (LLMs) has primarily focused on access-level behaviors such as refusals and safety filtering. However, equitable access does not ensure equitable interaction quality once a response is provided. In this paper, we conduct a controlled fairness audit examining how LLMs differ in tone, uncertainty, and linguistic framing across demographic identities after access is granted. Using a counterfactual prompt design, we evaluate GPT-4 and LLaMA-3.1-70B on career advice tasks while varying identity attributes along age, gender, and nationality. We assess access fairness through refusal analysis and measure interaction quality using automated linguistic metrics, including sentiment, politeness, and hedging. Identity-conditioned differences are evaluated using paired statistical tests. Both models exhibit zero refusal rates across all identities, indicating uniform access. Nevertheless, we observe systematic, model-specific disparities in interaction quality: GPT-4 expresses significantly higher hedging toward younger male users, while LLaMA exhibits broader sentiment variation across identity groups. These results show that fairness disparities can persist at the interaction level even when access is equal, motivating evaluation beyond refusal-based audits.