# Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?

arXiv: https://arxiv.org/abs/2210.14699 · HTML: https://mlsanity.com/paper/2210.14699/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2022.

## Built on by (most cited first, 14 of 14)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Next-Step Hint Generation for Introductory Programming Using Large Language Models](https://mlsanity.com/paper/2312.10055/) (2023)
- [HPC-Coder: Modeling Parallel Programs using Large Language Models](https://mlsanity.com/paper/2306.17281/) (2023)
- [Enhancing Automated Program Repair through Fine-tuning and Prompt Engineering](https://mlsanity.com/paper/2304.07840/) (2023)
- [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023)
- [Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models](https://mlsanity.com/paper/2301.13382/) (2023)
- [Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence](https://mlsanity.com/paper/2410.15490/) (2024)
- [Substance Beats Style: Why Beginning Students Fail to Code with LLMs](https://mlsanity.com/paper/2410.19792/) (2024)
- [GitHub Copilot: the perfect Code compLeeter?](https://mlsanity.com/paper/2406.11326/) (2024)
- [Automating Database-Native Function Code Synthesis with LLMs](https://mlsanity.com/paper/2604.06231/) (2026)
- [AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation](https://mlsanity.com/paper/2601.04540/) (2026)
- [Unlocking LLM Code Correction with Iterative Feedback Loops](https://mlsanity.com/paper/2606.17514/) (2026)
- [Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMs](https://mlsanity.com/paper/2503.16144/) (2025)
- [Evaluating the Use of LLMs for Automated DOM-Level Resolution of Web Performance Issues](https://mlsanity.com/paper/2601.05502/) (2026)

## Abstract (arXiv)

Language models are promising solutions for tackling increasing complex problems. In software engineering, they recently gained attention in code assistants, which generate programs from a natural language task description (prompt). They have the potential to save time and effort but remain poorly understood, limiting their optimal use. In this article, we investigate the impact of input variations on two configurations of a language model, focusing on parameters such as task description, surrounding context, model creativity, and the number of generated solutions. We design specific operators to modify these inputs and apply them to three LLM-based code assistants (Copilot, Codex, StarCoder2) and two benchmarks representing algorithmic problems (HumanEval, LeetCode). Our study examines whether these variations significantly affect program quality and how these effects generalize across models. Our results show that varying input parameters can greatly improve performance, achieving up to 79.27% success in one-shot generation compared to 22.44% for Codex and 31.1% for Copilot in default settings. Actioning this potential in practice is challenging due to the complex interplay in our study - the optimal settings for temperature, prompt, and number of generated solutions vary by problem. Reproducing our study with StarCoder2 confirms these findings, indicating they are not model-specific. We also uncover surprising behaviors (e.g., fully removing the prompt can be effective), revealing model brittleness and areas for improvement.