# AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?

arXiv: https://arxiv.org/abs/2312.10833 · HTML: https://mlsanity.com/paper/2312.10833/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2023.

## Built on by (most cited first, 8 of 8)

- [Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education](https://mlsanity.com/paper/2401.00832/) (2024)
- [Gemini Pro Defeated by GPT-4V: Evidence from Education](https://mlsanity.com/paper/2401.08660/) (2023)
- [Transforming Teachers' Roles and Agencies in the Era of Generative AI: Perceptions, Acceptance, Knowledge, and Practices](https://mlsanity.com/paper/2410.03018/) (2024)
- [How far can bias go? Tracing bias from pretraining data to alignment](https://mlsanity.com/paper/2411.19240/) (2024)
- [Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments](https://mlsanity.com/paper/2312.15842/) (2023)
- [BadFair: Backdoored Fairness Attacks with Group-conditioned Triggers](https://mlsanity.com/paper/2410.17492/) (2024)
- [Artificial Intelligence Bias on English Language Learners in Automatic Scoring](https://mlsanity.com/paper/2505.10643/) (2025)
- [G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German](https://mlsanity.com/paper/2402.06584/) (2024)

## Abstract (arXiv)

This study delves into the pervasive issue of gender issues in artificial intelligence (AI), specifically within automatic scoring systems for student-written responses. The primary objective is to investigate the presence of gender biases, disparities, and fairness in generally targeted training samples with mixed-gender datasets in AI scoring outcomes. Utilizing a fine-tuned version of BERT and GPT-3.5, this research analyzes more than 1000 human-graded student responses from male and female participants across six assessment items. The study employs three distinct techniques for bias analysis: Scoring accuracy difference to evaluate bias, mean score gaps by gender (MSG) to evaluate disparity, and Equalized Odds (EO) to evaluate fairness. The results indicate that scoring accuracy for mixed-trained models shows an insignificant difference from either male- or female-trained models, suggesting no significant scoring bias. Consistently with both BERT and GPT-3.5, we found that mixed-trained models generated fewer MSG and non-disparate predictions compared to humans. In contrast, compared to humans, gender-specifically trained models yielded larger MSG, indicating that unbalanced training data may create algorithmic models to enlarge gender disparities. The EO analysis suggests that mixed-trained models generated more fairness outcomes compared with gender-specifically trained models. Collectively, the findings suggest that gender-unbalanced data do not necessarily generate scoring bias but can enlarge gender disparities and reduce scoring fairness.