# Artificial Intelligence Bias on English Language Learners in Automatic Scoring

arXiv: https://arxiv.org/abs/2505.10643 · HTML: https://mlsanity.com/paper/2505.10643/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 15 May 2025.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT detectors are biased against non-native English writers](https://mlsanity.com/paper/2304.02819/) (2023)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Using Generative AI and Multi-Agents to Provide Automatic Feedback](https://mlsanity.com/paper/2411.07407/) (2024)
- [AI Gender Bias, Disparities, and Fairness: Does Training Data Matter?](https://mlsanity.com/paper/2312.10833/) (2023)
- [AI and Machine Learning for Next Generation Science Assessments](https://mlsanity.com/paper/2405.06660/) (2024)

## Built on by (most cited first, 3 of 3)

- [AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition](https://mlsanity.com/paper/2509.21910/) (2025)
- [Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading](https://mlsanity.com/paper/2601.22386/) (2026)
- [BRIDGE the Gap: Mitigating Bias Amplification in Automated Scoring of English Language Learners via Inter-group Data Augmentation](https://mlsanity.com/paper/2602.23580/) (2026)

## Abstract (arXiv)

This study investigated potential scoring biases and disparities toward English Language Learners (ELLs) when using automatic scoring systems for middle school students' written responses to science assessments. We specifically focus on examining how unbalanced training data with ELLs contributes to scoring bias and disparities. We fine-tuned BERT with four datasets: responses from (1) ELLs, (2) non-ELLs, (3) a mixed dataset reflecting the real-world proportion of ELLs and non-ELLs (unbalanced), and (4) a balanced mixed dataset with equal representation of both groups. The study analyzed 21 assessment items: 10 items with about 30,000 ELL responses, five items with about 1,000 ELL responses, and six items with about 200 ELL responses. Scoring accuracy (Acc) was calculated and compared to identify bias using Friedman tests. We measured the Mean Score Gaps (MSGs) between ELLs and non-ELLs and then calculated the differences in MSGs generated through both the human and AI models to identify the scoring disparities. We found that no AI bias and distorted disparities between ELLs and non-ELLs were found when the training dataset was large enough (ELL = 30,000 and ELL = 1,000), but concerns could exist if the sample size is limited (ELL = 200).