mlsanityData to 14 Oct 2026

Code Roulette: How Prompt Variability Affects LLM Code Generation

arXiv 2506.10204 ↗First posted 11 Jun 2025cs.SE cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Code generation is one of the most active areas of application of Large Language Models (LLMs). While LLMs lower barriers to writing code and accelerate development process, the overall quality of generated programs depends on the quality of given prompts. Specifically, functionality and quality of generated code can be sensitive to user's background and familiarity with software development. It is therefore important to quantify LLM's sensitivity to variations in the input. To this end we propose an evaluation pipeline for LLM code generation with a focus on measuring sensitivity to prompt augmentations, completely agnostic to a specific programming tasks and LLMs, and thus widely applicable. We provide extensive experimental evidence illustrating utility of our method and share our code for the benefit of the community.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 40 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

19 of 40 references.

References · 19

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  5. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  6. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  7. 2024EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluationscited by 34
  8. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  9. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  10. 2024Prompt Perturbation in Retrieval-Augmented Generation based Large Language Modelscited by 18
  11. 2024Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teamscited by 16
  12. 2024CS1-LLM: Integrating LLMs into CS1 Instructioncited by 16
7 more
  1. 2024The Impact of Prompt Programming on Function-Level Code Generationcited by 13
  2. 2023Testing LLMs on Code Generation with Varying Levels of Prompt Specificitycited by 12
  3. 2025Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolutioncited by 11
  4. 2025CodeArena: A Collective Evaluation Platform for LLM Code Generationcited by 10
  5. 2024Requirements are All You Need: The Final Frontier for End-User Software Engineeringcited by 5
  6. 2022Synthesizing Research on Programmers' Mental Models of Programs, Tasks and Concepts -- a Systematic Literature Reviewcited by 3
  7. 2025LLM Performance for Code Generation on Noisy Taskscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Paraphrasing Adversarial Attack on LLM-as-a-Reviewercited by 8
  2. 2026Within-Model vs Between-Prompt Variability in Large Language Models for Creative Taskscited by 3
  3. 2026Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoningcited by 0
  4. 2026Noisy Test-Time Reinforcement Learning for Code LLMscited by 0