# Bias Unveiled: Investigating Social Bias in LLM-Generated Code

arXiv: https://arxiv.org/abs/2411.10351 · HTML: https://mlsanity.com/paper/2411.10351/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2024.

## Built on by (most cited first, 25 of 25)

- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [The Bias is in the Details: An Assessment of Cognitive Bias in LLMs](https://mlsanity.com/paper/2509.22856/) (2025)
- [FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks](https://mlsanity.com/paper/2501.05396/) (2025)
- [Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs](https://mlsanity.com/paper/2506.11415/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering](https://mlsanity.com/paper/2604.16790/) (2026)
- [From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences](https://mlsanity.com/paper/2604.10332/) (2026)
- [BabelCoder: Agentic Code Translation with Specification Alignment](https://mlsanity.com/paper/2512.06902/) (2025)
- [Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration](https://mlsanity.com/paper/2512.20159/) (2025)
- [Social Bias in LLM-Generated Code: Benchmark and Mitigation](https://mlsanity.com/paper/2605.00382/) (2026)
- [Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code](https://mlsanity.com/paper/2605.05267/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectives](https://mlsanity.com/paper/2503.13793/) (2025)
- [Large Language Models' Varying Accuracy in Recognizing Risk-Promoting and Health-Supporting Sentiments in Public Health Discourse: The Cases of HPV Vaccination and Heated Tobacco Products](https://mlsanity.com/paper/2507.04364/) (2025)
- [GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Models](https://mlsanity.com/paper/2510.01252/) (2025)
- [What AI Speaks for Your Community: Polling AI Agents for Public Opinion on Data Center Projects](https://mlsanity.com/paper/2511.22037/) (2025)
- [A Multifaceted Analysis of Social Biases in Large Language Models](https://mlsanity.com/paper/2512.15792/) (2025)
- [Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Review](https://mlsanity.com/paper/2604.13103/) (2026)
- [From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation](https://mlsanity.com/paper/2604.21716/) (2026)
- [Large Language Models in K-12 Education: Alignment with State Curriculum Standards and Student Personas](https://mlsanity.com/paper/2606.04846/) (2026)
- [Wait, am I Being Fair? Characterizing Deductive Stereotyping and Mitigating It with Fair-GCG](https://mlsanity.com/paper/2606.30989/) (2026)
- [Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation](https://mlsanity.com/paper/2608.03691/) (2026)
- [GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI](https://mlsanity.com/paper/2608.21928/) (2026)
- [When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation](https://mlsanity.com/paper/2609.22102/) (2026)
- [A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Code](https://mlsanity.com/paper/2609.30642/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have significantly advanced the field of automated code generation. However, a notable research gap exists in evaluating social biases that may be present in the code produced by LLMs. To solve this issue, we propose a novel fairness framework, i.e., Solar, to assess and mitigate the social biases of LLM-generated code. Specifically, Solar can automatically generate test cases for quantitatively uncovering social biases of the auto-generated code by LLMs. To quantify the severity of social biases in generated code, we develop a dataset that covers a diverse set of social problems. We applied Solar and the crafted dataset to four state-of-the-art LLMs for code generation. Our evaluation reveals severe bias in the LLM-generated code from all the subject LLMs. Furthermore, we explore several prompting strategies for mitigating bias, including Chain-of-Thought (CoT) prompting, combining positive role-playing with CoT prompting and dialogue with Solar. Our experiments show that dialogue with Solar can effectively reduce social bias in LLM-generated code by up to 90%. Last, we make the code and data publicly available is highly extensible to evaluate new social problems.