mlsanityData to 14 Oct 2026

A Simple, Yet Effective Approach to Finding Biases in Code Generation

arXiv 2211.00609 ↗First posted 31 Oct 2022cs.AI cs.PLcited by 8

Not on the map. Why?

Abstract (arXiv)

Recently, high-performing code generation systems based on large language models have surfaced. They are trained on massive corpora containing much more natural text than actual executable computer code. This work shows that current code generation systems exhibit undesired biases inherited from their large language model backbones, which can reduce the quality of the generated code under specific circumstances. To investigate the effect, we propose the "block of influence" concept, which enables a modular decomposition and analysis of the coding challenges. We introduce an automated intervention mechanism reminiscent of adversarial testing that exposes undesired biases through the failure modes of the models under test. Finally, we demonstrate how our framework can be used as a data transformation technique during fine-tuning, acting as a mitigation strategy for these biases.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

34 of 54 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
22 more
  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  4. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  5. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  6. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  7. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  8. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  9. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  10. 2019SPoC: Search-based Pseudocode to Codecited by 176
  11. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  12. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  13. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  14. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 109
  15. 2022On the Paradox of Learning to Reason from Datacited by 93
  16. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  17. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  18. 2019Program Synthesis and Semantic Parsing with Learned Code Idiomscited by 34
  19. 2019Learning and Evaluating Contextual Embedding of Source Codecited by 33
  20. 2021LM-Critic: Language Models for Unsupervised Grammatical Error Correctioncited by 33
  21. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  22. 2022Measuring CLEVRness: Blackbox testing of Visual Reasoning Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 3; 2026: 2.

8 papers.

  1. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
  2. 2023Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMscited by 19
  3. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  4. 2025The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generationcited by 10
  5. 2026What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistantscited by 4
  6. 2025MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Taskscited by 1
  7. 2025Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Indexcited by 0
  8. 2026How Reasoning Shapes Social Bias in LLM-Generated Code?cited by 0