# A Multi-Language Perspective on the Robustness of LLM Code Generation

arXiv: https://arxiv.org/abs/2504.19108 · HTML: https://mlsanity.com/paper/2504.19108/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2025.

## Built on by (most cited first, 6 of 6)

- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [Defective Task Descriptions in LLM-Based Code Generation: Detection and Analysis](https://mlsanity.com/paper/2604.24703/) (2026)
- [Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs](https://mlsanity.com/paper/2605.29737/) (2026)
- [BabelCoder: Agentic Code Translation with Specification Alignment](https://mlsanity.com/paper/2512.06902/) (2025)
- [Social Bias in LLM-Generated Code: Benchmark and Mitigation](https://mlsanity.com/paper/2605.00382/) (2026)
- [The Biomimetic Architecture of Software 4.0](https://mlsanity.com/paper/2606.04025/) (2026)

## Abstract (arXiv)

Large language models have gained significant traction and popularity in recent times, extending their usage to code-generation tasks. While this field has garnered considerable attention, the exploration of testing and evaluating the robustness of code generation models remains an ongoing endeavor. Previous studies have primarily focused on code generation models specifically for the Python language, overlooking other widely used programming languages. In this work, we conduct a comprehensive comparative analysis to assess the robustness performance of several prominent code generation models and investigate whether robustness can be improved by repairing perturbed docstrings using an LLM. Furthermore, we investigate how their performance varies across different programming languages. To accomplish this, we introduce perturbations in four key areas of the prompt: DocString, function name, syntax, and format. We have compiled and released a dedicated dataset for this purpose. Our results show that all models consistently degrade under perturbations across all three languages, but vary in magnitude depending on the language and perturbation type. Larger model size does not reliably improve robustness, and semantic perturbations prove at least as disruptive as syntactic ones. Our LLM-based docString repair yields only marginal gains for simple perturbations and can degrade performance for semantic ones, highlighting the limits of prompt-level mitigation.