# Code Roulette: How Prompt Variability Affects LLM Code Generation

arXiv: https://arxiv.org/abs/2506.10204 · HTML: https://mlsanity.com/paper/2506.10204/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 19 of 19)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Out of the BLEU: how should we assess quality of the Code Generation models?](https://mlsanity.com/paper/2208.03133/) (2022)
- [EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations](https://mlsanity.com/paper/2410.22821/) (2024)
- [Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance](https://mlsanity.com/paper/2404.08817/) (2024)
- [NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations](https://mlsanity.com/paper/2406.19783/) (2024)
- [Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models](https://mlsanity.com/paper/2402.07179/) (2024)
- [Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams](https://mlsanity.com/paper/2402.17721/) (2024)
- [CS1-LLM: Integrating LLMs into CS1 Instruction](https://mlsanity.com/paper/2406.15379/) (2024)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [Testing LLMs on Code Generation with Varying Levels of Prompt Specificity](https://mlsanity.com/paper/2311.07599/) (2023)
- [Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution](https://mlsanity.com/paper/2501.11709/) (2025)
- [CodeArena: A Collective Evaluation Platform for LLM Code Generation](https://mlsanity.com/paper/2503.01295/) (2025)
- [Requirements are All You Need: The Final Frontier for End-User Software Engineering](https://mlsanity.com/paper/2405.13708/) (2024)
- [Synthesizing Research on Programmers' Mental Models of Programs, Tasks and Concepts -- a Systematic Literature Review](https://mlsanity.com/paper/2212.07763/) (2022)
- [LLM Performance for Code Generation on Noisy Tasks](https://mlsanity.com/paper/2505.23598/) (2025)

## Built on by (most cited first, 4 of 4)

- [Paraphrasing Adversarial Attack on LLM-as-a-Reviewer](https://mlsanity.com/paper/2601.06884/) (2026)
- [Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks](https://mlsanity.com/paper/2601.21339/) (2026)
- [Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning](https://mlsanity.com/paper/2608.30686/) (2026)
- [Noisy Test-Time Reinforcement Learning for Code LLMs](https://mlsanity.com/paper/2609.32172/) (2026)

## Abstract (arXiv)

Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose an evaluation pipeline for LLM code generation with a focus on measuring sensitivity to prompt augmentations, completely agnostic to a specific programming tasks and LLMs, and thus widely applicable. We provide extensive experimental evidence illustrating utility of our method and share our code for the benefit of the community.