# Metamorphic Testing for Fairness Evaluation in Large Language Models: Identifying Intersectional Bias in LLaMA and GPT

arXiv: https://arxiv.org/abs/2504.07982 · HTML: https://mlsanity.com/paper/2504.07982/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Does Gender Matter? Towards Fairness in Dialogue Systems](https://mlsanity.com/paper/1910.10486/) (2019)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [Quality Assurance of Bioinformatics Software: A Case Study of Testing a Biomedical Text Processing Tool Using Metamorphic Testing](https://mlsanity.com/paper/1802.07354/) (2018)

## Built on by (most cited first, 4 of 4)

- [Fairness Testing in Retrieval-Augmented Generation: How Small Perturbations Reveal Bias in Small Language Models](https://mlsanity.com/paper/2509.26584/) (2025)
- [LLM Assisted Coding with Metamorphic Specification Mutation Agent](https://mlsanity.com/paper/2511.18249/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)
- [No-Free-Fairness: Fundamental Limits and Trade-offs in Learning Systems](https://mlsanity.com/paper/2606.17810/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have made significant strides in Natural Language Processing but remain vulnerable to fairness-related issues, often reflecting biases inherent in their training data. These biases pose risks, particularly when LLMs are deployed in sensitive areas such as healthcare, finance, and law. This paper introduces a metamorphic testing approach to systematically identify fairness bugs in LLMs. We define and apply a set of fairness-oriented metamorphic relations (MRs) to assess the LLaMA and GPT model, a state-of-the-art LLM, across diverse demographic inputs. Our methodology includes generating source and follow-up test cases for each MR and analyzing model responses for fairness violations. The results demonstrate the effectiveness of MT in exposing bias patterns, especially in relation to tone and sentiment, and highlight specific intersections of sensitive attributes that frequently reveal fairness faults. This research improves fairness testing in LLMs, providing a structured approach to detect and mitigate biases and improve model robustness in fairness-sensitive applications.