# Exploring Multi-Lingual Bias of Large Code Models in Code Generation

arXiv: https://arxiv.org/abs/2404.19368 · HTML: https://mlsanity.com/paper/2404.19368/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2024.

## Built on by (most cited first, 17 of 17)

- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications](https://mlsanity.com/paper/2503.01245/) (2025)
- [DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation](https://mlsanity.com/paper/2408.13204/) (2024)
- [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025)
- [API-guided Dataset Synthesis to Finetune Large Code Models](https://mlsanity.com/paper/2408.08343/) (2024)
- [SEAD: Self-Evolving Agent for Multi-Turn Service Dialogue](https://mlsanity.com/paper/2602.03548/) (2026)
- [Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs](https://mlsanity.com/paper/2506.17353/) (2025)
- [RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry](https://mlsanity.com/paper/2505.15179/) (2025)
- [Reasoning as a Resource: Optimizing Fast and Slow Thinking in Code Generation Models](https://mlsanity.com/paper/2506.09396/) (2025)
- [CODECLEANER: Elevating Standards with A Robust Data Contamination Mitigation Toolkit](https://mlsanity.com/paper/2411.10842/) (2024)
- [A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics](https://mlsanity.com/paper/2505.15469/) (2025)
- [SEER: Enhancing Chain-of-Thought Code Generation through Self-Exploring Deep Reasoning](https://mlsanity.com/paper/2510.17130/) (2025)
- [TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Framework](https://mlsanity.com/paper/2510.17163/) (2025)
- [From If-Statements to ML Pipelines: Revisiting Bias in Code-Generation](https://mlsanity.com/paper/2604.21716/) (2026)
- [LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs](https://mlsanity.com/paper/2608.06041/) (2026)
- [Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture](https://mlsanity.com/paper/2608.28641/) (2026)
- [CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?](https://mlsanity.com/paper/2609.37216/) (2026)

## Abstract (arXiv)

Code generation aims to synthesize code and fulfill functional requirements based on natural language (NL) specifications, which can greatly improve development efficiency. In the era of large language models (LLMs), large code models (LCMs) have been recently proposed to generate source code. LCMs can generate highly feasible solutions for programming problems described in natural language. Despite the effectiveness, we observe a noticeable multilingual bias in the generation performance of LCMs. Specifically, LCMs demonstrate proficiency in generating solutions when provided with instructions in English, yet may falter when faced with semantically equivalent instructions in other NLs such as Chinese. Moreover, the ability of LCMs to generate code exhibits variety across different programming languages (PLs), such as Python and C++. The observed phenomenon indicates the presence of multi-lingual bias within the generative capabilities of LCMs, which has remained unexplored. In this paper, we aim to investigate the multi-lingual bias that exists in current LCMs. First, we initiate our investigation by constructing the first multi-lingual evaluation benchmark X-HumanEval-X, enabling us to systematically evaluate the extent of multi-lingual bias that exists in current LCMs. In our large-scale experiments on nine popular LCMs, we observe a pronounced multi-lingual bias of LCMs in code generation, including multi-NL and multi-PL bias. Specifically, when using Chinese instructions, the code generation capabilities of LCMs decrease by at least 13% in terms of the Pass@1 metric. Furthermore, LCMs perform variously across different programming languages, e.g., the performance gap between Python and C++ reaches as high as 20.9%. ...