# Testing LLMs on Code Generation with Varying Levels of Prompt Specificity

arXiv: https://arxiv.org/abs/2311.07599 · HTML: https://mlsanity.com/paper/2311.07599/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 10 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)

## Built on by (most cited first, 12 of 12)

- [What's Wrong with Your Code Generated by Large Language Models? An Extensive Study](https://mlsanity.com/paper/2407.06153/) (2024)
- [Benchmarking Prompt Sensitivity in Large Language Models](https://mlsanity.com/paper/2502.06065/) (2025)
- [Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation](https://mlsanity.com/paper/2407.05437/) (2024)
- [Tabular Embedding Model (TEM): Finetuning Embedding Models For Tabular RAG Applications](https://mlsanity.com/paper/2405.01585/) (2024)
- [Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generation](https://mlsanity.com/paper/2505.09027/) (2025)
- [Why is prompting hard? Understanding prompts on binary sequence predictors](https://mlsanity.com/paper/2502.10760/) (2025)
- [Code Roulette: How Prompt Variability Affects LLM Code Generation](https://mlsanity.com/paper/2506.10204/) (2025)
- [LOCOFY Large Design Models -- Design to code conversion solution](https://mlsanity.com/paper/2507.16208/) (2025)
- [Iterative Learning of Computable Phenotypes for Treatment Resistant Hypertension using Large Language Models](https://mlsanity.com/paper/2508.05581/) (2025)
- [Estimating the Error of Large Language Models at Pairwise Text Comparison](https://mlsanity.com/paper/2510.22219/) (2025)
- [Lifecycle-Aware code generation: Leveraging Software Engineering Phases in LLMs](https://mlsanity.com/paper/2510.24019/) (2025)
- [Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality](https://mlsanity.com/paper/2607.14816/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated unparalleled prowess in mimicking human-like text generation and processing. Among the myriad of applications that benefit from LLMs, automated code generation is increasingly promising. The potential to transform natural language prompts into executable code promises a major shift in software development practices and paves the way for significant reductions in manual coding efforts and the likelihood of human-induced errors. This paper reports the results of a study that evaluates the performance of various LLMs, such as Bard, ChatGPT-3.5, ChatGPT-4, and Claude-2, in generating Python for coding problems. We focus on how levels of prompt specificity impact the accuracy, time efficiency, and space efficiency of the generated code. A benchmark of 104 coding problems, each with four types of prompts with varying degrees of tests and specificity, was employed to examine these aspects comprehensively. Our results indicate significant variations in performance across different LLMs and prompt types, and its key contribution is to reveal the ideal prompting strategy for creating accurate Python functions. This study lays the groundwork for further research in LLM capabilities and suggests practical implications for utilizing LLMs in automated code generation tasks and test-driven development.