Explanation Fairness in Large Language Models: An Empirical Analysis of Disparities in How LLMs Justify Decisions Across Demographic Groups
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are increasingly deployed not only to make decisions but to explain them. While AI decision fairness has been studied extensively, the fairness of AI explanations (whether LLMs justify decisions with equal quality, depth, tone, and linguistic sophistication across demographic groups) has received little attention. This paper introduces the Explanation Fairness Taxonomy (EFT), a framework comprising five formally defined, operationalizable dimensions: Verbosity Disparity, Sentiment Disparity, Epistemic Hedging Disparity, Decision-Linked Explanation Disparity, and Lexical Complexity Disparity. The taxonomy is instantiated in a controlled empirical study across 80 prompt templates, four consequential decision domains (hiring, medical triage, credit assessment, legal judgment), and five LLMs: GPT-4.1, Claude Sonnet, LLaMA 3.3 70B, GPT-OSS 120B, and Qwen3 32B. Two novel black-box metrics are introduced: the Hedging Density Score (HDS) and the Explanation Faithfulness Proxy (EFP), a heuristic indicator of decision-linked explanation variation. Across up to 400 prompt pairs, all eight EFT metrics show statistically significant disparities (Cohen's d ranging from small to large, all p_BH < 10^(-62)). Model choice is strongly associated with disparity magnitude: Qwen3 32B exhibits verbosity disparities 5.9x larger than LLaMA 3.3 70B. Two prompting-based mitigations show significant reductions in EFP disparity (78-95%) but no significant effect on stylistic dimensions, consistent with the hypothesis that stylistic explanation inequalities are encoded in pre-training distributions and are not resolvable through deployment-level instruction alone. A reproducible measurement framework is offered for explanation-level fairness auditing, with implications for AI regulation and deployment practice.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 9 May 2026.
Builds on
17 of 26 references.
References · 17
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
- 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
- 2022The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanationscited by 41
5 more
- 2022Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanationscited by 40
- 2025Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarioscited by 25
- 2022Fairness and Explainability: Bridging the Gap Towards Fair Model Explanationscited by 22
- 2024Understanding Disparities in Post Hoc Machine Learning Explanationcited by 6
- 2026Equal Access, Unequal Interaction: A Counterfactual Audit of LLM Fairnesscited by 4
Built on by
Not cited yet.