# Unraveling Downstream Gender Bias from Large Language Models: A Study on AI Educational Writing Assistance

arXiv: https://arxiv.org/abs/2311.03311 · HTML: https://mlsanity.com/paper/2311.03311/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 6 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [50 Years of Test (Un)fairness: Lessons for Machine Learning](https://mlsanity.com/paper/1811.10104/) (2018)
- [Capturing Failures of Large Language Models via Human Cognitive Biases](https://mlsanity.com/paper/2202.12299/) (2022)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [Evaluation of Fairness Trade-offs in Predicting Student Success](https://mlsanity.com/paper/2007.00088/) (2020)
- [Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modeling](https://mlsanity.com/paper/2209.10335/) (2022)
- [The Undesirable Dependence on Frequency of Gender Bias Metrics Based on Word Embeddings](https://mlsanity.com/paper/2301.00792/) (2023)

## Built on by (most cited first, 15 of 15)

- [Gender Bias in Large Language Models across Multiple Languages](https://mlsanity.com/paper/2403.00277/) (2024)
- [TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students](https://mlsanity.com/paper/2410.04078/) (2024)
- [Towards Modeling Learner Performance with Large Language Models](https://mlsanity.com/paper/2403.14661/) (2024)
- [Biased AI can Influence Political Decision-Making](https://mlsanity.com/paper/2410.06415/) (2024)
- [Modifying AI, Enhancing Essays: How Active Engagement with Generative AI Boosts Writing Quality](https://mlsanity.com/paper/2412.07200/) (2024)
- [Eagle: Ethical Dataset Given from Real Interactions](https://mlsanity.com/paper/2402.14258/) (2024)
- [Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language](https://mlsanity.com/paper/2507.16557/) (2025)
- [Identifying Fairness Issues in Automatically Generated Testing Content](https://mlsanity.com/paper/2404.15104/) (2024)
- [Anna Karenina Strikes Again: Pre-Trained LLM Embeddings May Favor High-Performing Learners](https://mlsanity.com/paper/2406.06599/) (2024)
- [Social Bias in Multilingual Language Models: A Survey](https://mlsanity.com/paper/2508.20201/) (2025)
- [Controllable Pareto Trade-off between Fairness and Accuracy](https://mlsanity.com/paper/2509.13651/) (2025)
- [A Multifaceted Analysis of Social Biases in Large Language Models](https://mlsanity.com/paper/2512.15792/) (2025)
- [Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writing](https://mlsanity.com/paper/2606.15914/) (2026)
- [EduFair-Bench: Evaluating Pedagogical Fairness of LLM Tutors Across Student Demographics](https://mlsanity.com/paper/2609.12949/) (2026)
- [DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions](https://mlsanity.com/paper/2609.18649/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly utilized in educational tasks such as providing writing suggestions to students. Despite their potential, LLMs are known to harbor inherent biases which may negatively impact learners. Previous studies have investigated bias in models and data representations separately, neglecting the potential impact of LLM bias on human writing. In this paper, we investigate how bias transfers through an AI writing support pipeline. We conduct a large-scale user study with 231 students writing business case peer reviews in German. Students are divided into five groups with different levels of writing support: one classroom group with feature-based suggestions and four groups recruited from Prolific -- a control group with no assistance, two groups with suggestions from fine-tuned GPT-2 and GPT-3 models, and one group with suggestions from pre-trained GPT-3.5. Using GenBit gender bias analysis, Word Embedding Association Tests (WEAT), and Sentence Embedding Association Test (SEAT) we evaluate the gender bias at various stages of the pipeline: in model embeddings, in suggestions generated by the models, and in reviews written by students. Our results demonstrate that there is no significant difference in gender bias between the resulting peer reviews of groups with and without LLM suggestions. Our research is therefore optimistic about the use of AI writing support in the classroom, showcasing a context where bias in LLMs does not transfer to students' responses.