mlsanityData to 14 Oct 2026

Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations

arXiv 2509.13680 ↗First posted 17 Sep 2025cs.SE cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Code generation models are widely used in software development, yet their sensitivity to prompt phrasing remains under-examined. Identical requirements expressed with different emotions or communication styles can yield divergent outputs, while most benchmarks emphasize only peak performance. We present PromptSE (Prompt Sensitivity Evaluation), a framework that creates semantically equivalent prompt variants with emotion and personality templates, and that evaluates stability using probability aware continuous scoring or using binary pass rates when logits are unavailable. The results are aggregated into a proposed area under curve metric (AUC-E) for cross model comparison. Across 14 models from three families (Llama, Qwen, and DeepSeek), our study shows that performance and stability behave as largely decoupled optimization objectives, and it reveals architectural and scale related patterns that challenge common assumptions about model robustness. The framework supports rapid screening for closed-source models as well as detailed stability analysis in research settings. PromptSE enables practitioners to quantify performance stability trade offs for deployment and model selection, positioning prompt stability as a complementary evaluation dimension alongside performance and fairness, and contributing to more trustworthy AI-assisted software development tools.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

20 of 40 references.

References · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2019How Can We Know What Language Models Know?cited by 959
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  11. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  12. 2024Does Prompt Formatting Have Any Impact on LLM Performance?cited by 141
8 more
  1. 2025From Personas to Talks: Revisiting the Impact of Personas on LLM-Synthesized Emotional Support Conversationscited by 23
  2. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  3. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  4. 2025When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMscited by 10
  5. 2024Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMscited by 7
  6. 2023Emotion-Conditioned Text Generation through Automatic Prompt Optimizationcited by 6
  7. 2024CodeFort: Robust Training for Code Generation Modelscited by 4
  8. 2024Personality-Guided Code Generation Using Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.