mlsanityData to 7 Oct 2026

Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

arXiv 2607.14816 ↗First posted 16 Jul 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 100 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

49 of 100 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2024A Survey on Large Language Models for Code Generationcited by 646
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  10. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  11. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  12. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
37 more
  1. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  2. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  3. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  4. 2023Software Testing with Large Language Models: Survey, Landscape, and Visioncited by 163
  5. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 136
  6. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
  7. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 87
  8. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  9. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  10. 2023How Secure is Code Generated by ChatGPT?cited by 77
  11. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  12. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  13. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  14. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  15. 2022Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?cited by 63
  16. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 63
  17. 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 43
  18. 2023Towards Human-Bot Collaborative Software Architecting with ChatGPTcited by 31
  19. 2022ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languagescited by 29
  20. 2023Improving ChatGPT Prompt for Code Generationcited by 28
  21. 2023A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repaircited by 26
  22. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
  23. 2020Predicting Defective Lines Using a Model-Agnostic Techniquecited by 14
  24. 2023From Copilot to Pilot: Towards AI Supported Software Developmentcited by 14
  25. 2023A User-centered Security Evaluation of Copilotcited by 14
  26. 2023Testing LLMs on Code Generation with Varying Levels of Prompt Specificitycited by 12
  27. 2023The Program Testing Ability of Large Language Models for Codecited by 11
  28. 2024Optimizing Large Language Model Hyperparameters for Code Generationcited by 10
  29. 2024LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignmentscited by 6
  30. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attentioncited by 4
  31. 2025Is Quantization a Deal-breaker? Empirical Insights from Large Code Modelscited by 4
  32. 2020Task-Oriented API Usage Examples Prompting Powered By Programming Task Knowledge Graphcited by 3
  33. 2024Large Language Models for Code Summarizationcited by 3
  34. 2019Some SonarQube Issues have a Significant but SmallEffect on Faults and Changes. A large-scale empirical studycited by 1
  35. 2023Vulnerable Source Code Detection using SonarCloud Code Analysiscited by 1
  36. 2024Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilotcited by 1
  37. 2025Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMscited by 1

Built on by

Not cited yet.