mlsanityData to 14 Oct 2026

Uncovering and Quantifying Social Biases in Code Generation

arXiv 2305.15377 ↗First posted 24 May 2023cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

With the popularity of automatic code generation tools, such as Copilot, the study of the potential hazards of these tools is gaining importance. In this work, we explore the social bias problem in pre-trained code generation models. We propose a new paradigm to construct code prompts and successfully uncover social biases in code generation models. To quantify the severity of social biases in generated code, we develop a dataset along with three metrics to evaluate the overall social bias and fine-grained unfairness across different demographics. Experimental results on three pre-trained code generation models (Codex, InCoder, and CodeGen) with varying sizes, reveal severe social biases. Moreover, we conduct analysis to provide useful insights for further choice of code generation models with low social bias. (This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups.)

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

48 of 68 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  11. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  12. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
36 more
  1. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  3. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 463
  4. 2021Conditional DETR for Fast Training Convergencecited by 424
  5. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  6. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  7. 2019Evaluating Gender Bias in Machine Translationcited by 249
  8. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 224
  9. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 166
  10. 20203D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Priorcited by 126
  11. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  12. 2020Towards Controllable Biases in Language Generationcited by 112
  13. 2023Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinementcited by 106
  14. 2022Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignmentcited by 98
  15. 2021An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Modelscited by 97
  16. 2021Trustworthy AI: A Computational Perspectivecited by 93
  17. 2022Compressible-composable NeRF via Rank-residual Decompositioncited by 92
  18. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  19. 2020Multi-Dimensional Gender Bias Classificationcited by 80
  20. 2021Joint Implicit Image Function for Guided Depth Super-Resolutioncited by 31
  21. 2022MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentationcited by 30
  22. 2022Group DETR v2: Strong Object Detector with Encoder-Decoder Pretrainingcited by 25
  23. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  24. 2021Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspectivecited by 22
  25. 2022Conditional DETR V2: Efficient Detection Transformer with Box Queriescited by 21
  26. 2022Point Scene Understanding via Disentangled Instance Mesh Reconstructioncited by 20
  27. 2021The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classificationcited by 18
  28. 2022D3ETR: Decoder Distillation for Detection Transformercited by 17
  29. 2022Learning Fair Representations via Rate-Distortion Maximizationcited by 15
  30. 2021Toward Annotator Group Bias in Crowdsourcingcited by 14
  31. 2022On Measuring Social Biases in Prompt-Based Multi-Task Learningcited by 10
  32. 2022HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Modelscited by 8
  33. 2021Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarizationcited by 6
  34. 2022Gender Bias in Meta-Embeddingscited by 6
  35. 2023Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarioscited by 4
  36. 2023Real-time 3D Semantic Scene Completion Via Feature Aggregation and Conditioned Predictioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 8; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 658
  2. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  3. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  4. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  5. 2025Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applicationscited by 26
  6. 2025A Study of LLMs' Preferences for Libraries and Programming Languagescited by 20
  7. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 11
  8. 2025FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Taskscited by 9
  9. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  10. 2025Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Taskscited by 6
  11. 2025Mitigating Biases in Language Models via Bias Unlearningcited by 6
  12. 2023Uncovering and Categorizing Social Biases in Text-to-SQLcited by 5
13 more
  1. 2024Improving Long Text Understanding with Knowledge Distilled from Summarization Modelcited by 3
  2. 2024Mitigating Gender Bias in Code Large Language Models via Model Editingcited by 3
  3. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  4. 2026Social Bias in LLM-Generated Code: Benchmark and Mitigationcited by 1
  5. 2025Mapping the Trust Terrain: LLMs in Software Engineering -- Insights and Perspectivescited by 0
  6. 2025What Makes Code Generation Ethically Sourced?cited by 0
  7. 2026Fairness in Multi-Agent Systems for Software Engineering: An SDLC-Oriented Rapid Reviewcited by 0
  8. 2026From If-Statements to ML Pipelines: Revisiting Bias in Code-Generationcited by 0
  9. 2026Measuring and Mitigating Bias in Code Generated by Large Language Modelscited by 0
  10. 2026Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generationcited by 0
  11. 2026LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMscited by 0
  12. 2026When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generationcited by 0
  13. 2026A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Codecited by 0